psid64-devel Mailing List for PSID64

> In that case, may I propose:
>
> http://lala.c64.org/SID_file_format_v3b.zip

Great work! Here are my remarks:

The RSID format was introduced to let players reject certain SID
files. Actually it is more like another bit field saying 'real C64
environment required'. If we can get rid of the PSID/RSID
differentiation in V3 that would have my preference as it greatly
simplifies the format. I have not yet looked at the consequences of
such a decision. At least the flags field would require another bit:

+76    WORD flags
- Bit 1 specifies whether the tune is PlaySID specific, e.g. uses PlaySID
  samples (psidSpecific):
  0 =3D C64 compatible,
  1 =3D PlaySID specific (PSID v2NG, v3).
- Bit 6 specifies whether the song is a BASIC program (PSID v3).
  0 =3D Machine code program,
  1 =3D BASIC program.
- Bits 7-15 are reserved and should be set to 0.

> - Extended SPEED field with support for arbitrary number of songs (if nee=
ded).

Whether we still need this depends on my previous remark.

> This pretty much satisfies all of our requirements, except:
>
> - Can't specify license type (can be solved simply by adding a new field =
definition) (BTW, this requirement is pretty low on my priority list).

This can be added easily by introducing another field code, but only
if the authors see added value in the ability to specify the license
type. Maybe a legal expert can tell whether or not this is useful in
cases similar to the Acid Jazzed Evening incident.

> - Supports only a single C64 file.

Probably cannot be handled in a nice way without modifying PSID format
too much. I agree to mark it as WON'T FIX.

> - No ROM support.

This was mentioned by Simon White IIRC. I don't know the priority of this i=
tem.

> But it is fully backwards compatible since it's built on the existing PSI=
Dv2NG, it's easily extensible in the future with new fields if needed, C64 =
binary compatibility remains, etc.
>
> I hope this proposal provides a sound basis for a discussion of the detai=
ls.

OK, some details:

- Most binary formats I'm familiar with specify the length of a field
EXCLUDING the field length and field code bytes. A simple reason for
this is that the parser already has read that data and just needs to
read another <length> bytes to retrieve the actual field data.

- The extensions are quite hard to 'parse' manually in a hex editor.
With the current PSID v2 header I often just edited the fields in a
hex editor as SIDedit segfaults on my Fedora Core 6 system :-( (looks
like a bug in the Perl/Tk implementation). Did you consider using
FourCC (four character code) instead of just a single byte? I don't
mind sticking with the single byte field codes.

- I think we have to specify how to interpret the field of a SID file
if it appears multiple times, e.g. if it contains twice the author
field of song number 1. We can either use the LAST definition in such
a case or reject the entire SID file.

- Do not leave fields undefined that have no meaning in a specific
context. These fields are reserved within that context and MUST be set
to 0. Examples are SONGLENGTH_LOOP_POINT and SPEED_EXTENDED

> +16    ``<title>''
> +36    ``<author>''
> +56    ``<released>'' (also known as ``<copyright>'')
>
> NOTE: Version 3 provides alternative <title>, <author> and <released>
> fields that can be longer than 31 chars and use UTF-8 Unicode encoding.
> In such cases every effort should be made to make the version 2 fields
> roughly equivalent to the alternative ones.

Although not recommended should we support leaving such a field empty,
i.e. a string of length zero?

> SONGLENGTH_INFO +00 WORD <SONGLENGTH>

- What rounding is to be used for this field (floor, ceil, round)?

- Currently the longest song in the HVSC lasts 34:24, so 1:49:13 ought
to be enough for everyone :-). We could define 0xFFFF as Infinite.

> SONGLENGTH_INFO +02 BYTE <SONGLENGTH_REPEAT_FLAG>

Is there a need for this field? I know it's in the Songlengths.txt
file right now but who uses it and for what reason?

> SONGLENGTH_INFO +03 WORD <SONGLENGTH_LOOP_POINT>

- What rounding is to be used for this field (floor, ceil, round)?

- I propose to mandate that SONGLENGTH_LOOP_POINT is set to 0xFFFF for
songs that do not loop (matches nicely with defintion of 0xFFFF for
SONGLENGTH).

> 0x05 =96 SPEED_EXTENDED ... bit 0 in longword 1 will correspond to song n=
umber 64 ...

This should be song number 65, right?

--=20
Roland

> Another way to see it is that we have covered all issues and can
> continue to the next phase to prioritize the requirements and define
> the format.

In that case, may I propose:

http://lala.c64.org/SID_file_format_v3b.zip

Key points:

- Based on existing PSIDv2NG format spec.

- UTF8 Unicode TITLE and AUTHOR fields per file or per song (if needed - if not present, existing TITLE and AUTHOR fields can be used).

- UTF8 Unicode RELEASED field, one per file (if needed).

- UFT8 Unicode free-form COMMENT (or STIL INFO) field per file or per song (again, only if needed).

- Songlength info with length in 0.1 second resolution, song repeat flag and loop point if applicable.

- Extended SPEED field with support for arbitrary number of songs (if needed).


This pretty much satisfies all of our requirements, except:

- Can't specify license type (can be solved simply by adding a new field definition) (BTW, this requirement is pretty low on my priority list).
- Supports only a single C64 file.
- No ROM support.

But it is fully backwards compatible since it's built on the existing PSIDv2NG, it's easily extensible in the future with new fields if needed, C64 binary compatibility remains, etc.

I hope this proposal provides a sound basis for a discussion of the details.

--

Imre Olajos

> So, judging from the inactivity, I guess the answer to the question is "no new file format needed"?...

Another way to see it is that we have covered all issues and can
continue to the next phase to prioritize the requirements and define
the format.

-- 
Roland

LaLa schrieb:
> So, judging from the inactivity, I guess the answer to the question is "no new file format needed"?...

*wakes up, raises head*

Huh?

*eyelids slowly close again*

So, judging from the inactivity, I guess the answer to the question is "no new file format needed"?...
 
--

Imre Olajos

> > To answer your question, I think the ultimate decision lays in the hands of the HVSC Crew. If we come up with a new file format and the HVSC Crew decides not to adopt it, that's pretty much a death sentence. With no HVSC Crew support developers will also lack the incentive to upgrade their own SID players, HVSC tools, etc. to support a new format that is not used by the HVSC at all.

You couldn't have said it better. Support from the HVSC Crew is
crucial if we want a new (maybe *extended* is a better word in this
case) file format.

> > But I actually got excited by the prospect of a new file format! It was the spark I needed to get me excited about SID files again.

:-)

> Same here, I'm 100% with you on this. But note that I've declared my retreat
> from the HVSC admin position and also from the HVSC team after update #47, so my
> opinion is probably not really crucial anymore. If you don't mind I'd like to
> help out with shaping the new file format nevertheless. If that's the last
> service to the sid community I can offer, I'd be happy to do so.

Your valuable input is well appreciated. Is it already known who will
take over your admin position? As I already said before I'm willing to
help migrate exising HVSC tools and collection, but I would need some
input from the HVSC crew on what tools currently exist.

> PS: Roland, hitting "reply" doesn't include the mailing list addy, only the
> address of the original poster. Can this be fixed?

Should be better now. Apparently the default is to reply to the poster only.

Regards,
Roland

LaLa schrieb:
>> Seconds should really be precise enough, let's not be more precise than 
>> necessary, please.
> 
> I've heard others complain that the second-precision is not good enough. I tend to agree: especially with repeating tunes 0.9 seconds can make the difference between a smooth and a rough transition in playlists.

Ok, won't argue here, let's make it 10th of a second, fine.


>> Now for something different: Who actually is to decide about the fileformat? I'm 
>> just asking because I got some quite discouraging feedback from the crew, along 
>> the lines of "Nah, it worked ever since, why should we change" or "I like it the 
>> way it is", constructive stuff like that. So before we start investing too much 
>> of our time here I'd like to clear up who has the final say.
> 
> Ahh, blessed complacency...
> 
> Maybe if the crew was presented with an already agreed-on and polished new SID format, they'd think differently? Maybe if they were presented with a new update.exe and SIDedit supporting all the new fields, plus tools to convert the current HVSC to the new format, they'd think differently? =)
> 
> To answer your question, I think the ultimate decision lays in the hands of the HVSC Crew. If we come up with a new file format and the HVSC Crew decides not to adopt it, that's pretty much a death sentence. With no HVSC Crew support developers will also lack the incentive to upgrade their own SID players, HVSC tools, etc. to support a new format that is not used by the HVSC at all.
> 
> Believe me, I understand the Crew's complacency towards this. I suffer from the same thing with regards to SIDedit. Lack of time is a major factor for lack of development, but I was also struggling with adding support for songlengths and the like to SIDedit. I also got discouraged by the many bugs I've found in the Perl/Tk implementation during my development...
> 
> But I actually got excited by the prospect of a new file format! It was the spark I needed to get me excited about SID files again.
> 
> But that's just me, I guess...

Same here, I'm 100% with you on this. But note that I've declared my retreat
from the HVSC admin position and also from the HVSC team after update #47, so my
opinion is probably not really crucial anymore. If you don't mind I'd like to
help out with shaping the new file format nevertheless. If that's the last
service to the sid community I can offer, I'd be happy to do so.

Regards,
Stephan

PS: Roland, hitting "reply" doesn't include the mailing list addy, only the 
address of the original poster. Can this be fixed?

Steppe,

> Seconds should really be precise enough, let's not be more precise than 
> necessary, please.

I've heard others complain that the second-precision is not good enough. I tend to agree: especially with repeating tunes 0.9 seconds can make the difference between a smooth and a rough transition in playlists.

> Now for something different: Who actually is to decide about the fileformat? I'm 
> just asking because I got some quite discouraging feedback from the crew, along 
> the lines of "Nah, it worked ever since, why should we change" or "I like it the 
> way it is", constructive stuff like that. So before we start investing too much 
> of our time here I'd like to clear up who has the final say.

Ahh, blessed complacency...

Maybe if the crew was presented with an already agreed-on and polished new SID format, they'd think differently? Maybe if they were presented with a new update.exe and SIDedit supporting all the new fields, plus tools to convert the current HVSC to the new format, they'd think differently? =)

To answer your question, I think the ultimate decision lays in the hands of the HVSC Crew. If we come up with a new file format and the HVSC Crew decides not to adopt it, that's pretty much a death sentence. With no HVSC Crew support developers will also lack the incentive to upgrade their own SID players, HVSC tools, etc. to support a new format that is not used by the HVSC at all.

Believe me, I understand the Crew's complacency towards this. I suffer from the same thing with regards to SIDedit. Lack of time is a major factor for lack of development, but I was also struggling with adding support for songlengths and the like to SIDedit. I also got discouraged by the many bugs I've found in the Perl/Tk implementation during my development...

But I actually got excited by the prospect of a new file format! It was the spark I needed to get me excited about SID files again.

But that's just me, I guess...

--

LaLa

Hi LaLa,

LaLa schrieb:
>  > However I can imagine that putting all the infromation into the SID file,
>  > would seriously complicate things for the HVSC crew. Any slight change
>  > in the song credits would require assembling a SID file from the start,
>  > instead of just editing a text file. But I guess a tool could automate
>  > this task...

Roland, I can't agree here. Changing a single bit of info thus far is done by a 
simple command in the update script, so if you include more commands in that it 
doesn't get more complicated at all. Just adapt the script to cope with commands 
like these:

PLAYTIME
# auto sldb is far off by over a minute
/MUSICIANS/D/DRAX/Clarencio.sid
3:31 (G)

STILTITLE
# albuminfo added
/MUSICIANS/D/DRAX/Clarencio.sid
Clarence's Dream [from blablba]

It's absolutely the same in green. And with Lala's Sidedit updated at the same 
time it would really make no difference at all.


> Obviously, any kind of format switch will be painful for the HVSC crew 
> at first, but I think it would be worth it in the long run. Consider 
> this: currently, if just -one- STIL entry changes in HVSC, the entire 
> STIL.txt file still has to be included in an HVSC update. If STIL 
> entries were part of SID files, only that SID file would have to be 
> included. Same with the song length info. So, a new file format would 
> actually make the update process more efficient. Not to mention the 
> original reason that started this discussion: the desire to have access 
> to all this metadata without consulting those special files in the HVSC 
> all the time - sometimes that is just not feasible (say, on a real C64).

That sums it up pretty nicely. From an administrative point of view it's not 
much difference where you have your information stored. It's the way you access 
and change the info that matters.


>  > Also, I don't have a clue how the songlength calculation tool works, 
> but if it's
>  > capable of telling that a song is looped, than maybe it can also tell
>  > where is the loop-point. If it can (or it can be done) I would like 
> to see
>  > a field having the loop-point time apart from only the song's play time.
>  > Also both these values should be as accurate as possible. Say, stored 
> as a
>  > number of miliseconds or even better as as number of clock cycles or 
> timer
>  > ticks.

Actually the Songlengths db does say if a song loops or goes silent.
If the songlength entry says 2:20, it loops at 2:20, easy as that.
If it says 2:20(G), 2:20(Z), 2:20(M), 2:20(B), it doesn't, but goes silent at 
that time mark.


> Personally, I think milliseconds is an overkill for songlengths, but 
> maybe 1/10th of a second provides enough resolution without bloating the 
> file size. I would have a preference for seconds-based song length info, 
> because that's what listeners are really interested in. However, a new 
> file format could also store timer ticks or something like that in 
> parallel with seconds/milliseconds/etc.

Seconds should really be precise enough, let's not be more precise than 
necessary, please.

Now for something different: Who actually is to decide about the fileformat? I'm 
just asking because I got some quite discouraging feedback from the crew, along 
the lines of "Nah, it worked ever since, why should we change" or "I like it the 
way it is", constructive stuff like that. So before we start investing too much 
of our time here I'd like to clear up who has the final say.

Regards,
Stephan

> However I can imagine that putting all the infromation into the SID file,
> would seriously complicate things for the HVSC crew. Any slight change

> in the song credits would require assembling a SID file from the start,
> instead of just editing a text file. But I guess a tool could automate 
> this task...

For now I wouldn't be worried about how information will be changed in the new SID files - first we should concentrate on getting the format right . I know that quite a few people in the HVSC crew are still using my SIDedit tool - that could be extended to deal with a new file format, too. I also created an Audio::SID Perl module which is freely available in CPAN - that can also be changed to deal with a new file format, which would make it possible to write Perl scripts that would automate such tedious tasks as updating textual data in SID files. And I'm sure others would come up with tools and scripts to help with the administrative tediousness.

Obviously, any kind of format switch will be painful for the HVSC crew at first, but I think it would be worth it in the long run. Consider this: currently, if just -one- STIL entry changes in HVSC, the entire STIL.txt file still has to be included in an HVSC update. If STIL entries were part of SID files, only that SID file would have to be included. Same with the song length info. So, a new file format would actually make the update process more efficient. Not to mention the original reason that started this discussion: the desire to have access to all this metadata without consulting those special files in the HVSC all the time - sometimes that is just not feasible (say, on a real C64).

> Anyway, if you manage to enforce a new format please don't forget to take

> care of the "speed" value for subsongs above number 32. This constantly
> causes problems for non-RSID files.

Duly noted!

> Also, I don't have a clue how the songlength calculation tool works, but if it's

> capable of telling that a song is looped, than maybe it can also tell
> where is the loop-point. If it can (or it can be done) I would like to see
> a field having the loop-point time apart from only the song's play time. 

> Also both these values should be as accurate as possible. Say, stored as a
> number of miliseconds or even better as as number of clock cycles or timer 
> ticks.


Several great points here. I know the songlength calculator tool written by Michael Schwendt can tell if a song is looping or not - in the current Songlengths.txt file any songlength with (x) means that the tune is NOT looping (with 'x' denoting the reason why), any other entry is a looping song. However, I do not think the tool is smart enough to figure out where the loop point is. It can tell you that a tune is 2:00 long after which it repeats, but I do not think it can tell that it restarts at, say, 1:05, instead of 0:00. I imagine that would not be easy to do.

Personally, I think milliseconds is an overkill for songlengths, but maybe 1/10th of a second provides enough resolution without bloating the file size. I would have a preference for seconds-based song length info, because that's what listeners are really interested in. However, a new file format could also store timer ticks or something like that in parallel with seconds/milliseconds/etc.

--

LaLa

Hello there,

I agree than the song metadata is spread over a little bit too much.

However I can imagine that putting all the infromation into the SID file,
would seriously complicate things for the HVSC crew. Any slight change
in the song credits would require assembling a SID file from the start,
instead of just editing a text file. But I guess a tool could automate
this task...

Anyway, if you manage to enforce a new format please don't forget to take
care of the "speed" value for subsongs above number 32. This constantly
causes problems for non-RSID files.

Also, I don't have a clue how the songlength calculation tool works, but if
it's
capable of telling that a song is looped, than maybe it can also tell
where is the loop-point. If it can (or it can be done) I would like to see
a field having the loop-point time apart from only the song's play time.
Also both these values should be as accurate as possible. Say, stored as a
number of miliseconds or even better as as number of clock cycles or timer
ticks.

Cheers,
Sebastian

> > 8. SID file supports only a single C64 file.
>
> This requirement alone could force us to ditch the current PSID file format completely and start from scratch.

I know, and I put it there for completeness. If we all agree that it's
not that important (e.g. I consider it a "nice to have") and given the
impact it will have (huge), we can just decide not to implement it.

-- 
Roland

> Thanks for putting this together. To structure the process a bit I
> propose to focus first on the limitations of the current formats (PSID
> v2NG and RSID) and desired extensions.

Good thinking.

> 1. Meta data is spread over different files: the .sid file itself,
> STIL.txt, Songlengths.txt and BUGlist.txt.

I would argue that the format specification itself only has to provide per-subsong and per-file free-form comment fields. If HVSC wants to put specifically formatted content in those comment fields (say, STIL info, BUG info, etc.), it's up to the HVSC Team, but I would argue that it's not something that should be specified by the file format. This is similar to how a SID file can have any filename as long as it has the .sid extension, but the HVSC puts an additional restriction on the filename in that it must contain only POSIX compatible characters for cross-platform compatibility. That's a restriction imposed by the HVSC archive, not by the PSID file format spec.

> 2. HVSC uses two similar but yet different file formats: PSID V2NG and
> RSID. The formats interpret flag bit 1 differently (PlaySID specific
> vs C64 BASIC).

Well, there's a bit more to that, because RSID is basically PSID v2NG with additional restrictions and assumptions. See the "magicID" description in the file format spec.

> 8. SID file supports only a single C64 file.

This requirement alone could force us to ditch the current PSID file format completely and start from scratch.

--
LaLa

> To get the ball rolling, here is a quick draft of a proposal I am putting forward: an extension of the current PSID/RSID format with new v3 fields added to it. The document is in Microsoft Word so that it can show you the changes I made to the PSID v2NG file format description, but I also included a plain-text version.

Thanks for putting this together. To structure the process a bit I
propose to focus first on the limitations of the current formats (PSID
v2NG and RSID) and desired extensions. The issues I gathered up till
now:
1. Meta data is spread over different files: the .sid file itself,
STIL.txt, Songlengths.txt and BUGlist.txt.
2. HVSC uses two similar but yet different file formats: PSID V2NG and
RSID. The formats interpret flag bit 1 differently (PlaySID specific
vs C64 BASIC).
3. Text fields only support ISO Latin 1, which is used e.g. on Amiga.
Unicode UTF-8 character support is desired.
4. No ability to specify license type, e.g. creative commons,
commercial. See
http://en.wikipedia.org/wiki/Category:Copyright_licenses for more
license types. This would allow the authors to release their work
under a specific license.
5. Name, author and released fields are limited to 31 characters.
6. Release date has year resolution.
7. Speed bits for first 32 songs (may no longer be an issue).
8. SID file supports only a single C64 file.
9. Support for ROMs, e.g. Simons' BASIC.

Non-functional requirements:
1. Maintain backwards compatibility.
2. C64 usability. The data format can be processed easily on a real
C64, without requiring any hardware extensions.
3. Easily extensible, so new fields can be added in the future without
requiring a complete rewrite of the file format.

Regards,
Roland

> I do not know how rigorous other software developers were when implementing PSID loader routines, but technically, they should reject files if they a) don't start with either "PSID" or "RSID", b) have a version number larger than 2, c) have a dataOffset other than 0x0076 or 0x007C. I'm just thinking of backwards compatibility here.

By looking at e.g. TinySID source I noticed that it only uses the
least significant byte of the header length word, but I'm sure Rainer
Sinsch is willing fix this.

The original PlaySID code declares the dataOffset field as "length",
most likely short for "header length". One can either call the field
dataOffset and declare everything before this address part of the
header, or call the field headerLength and declare everything after it
data. In the end it's just a matter of wording and doesn't change the
contents of a file, but the latter case sounds more logical to me.

> One reason we switched the magic ID to "RSID" was so that non-RSID-compatible players would reject those files. We could do the same (say, use "XSID" for version > 2).

I guess that's the function of a version field ;-). In general
software should not expect forward compatibility of a file format and
thus not accept any newer versions than it can handle.

> BTW, add Unicode UTF-8 support to my list of desired features. So we could have longer and linguistically correct "title", "author" and "copyright" fields. (We could keep the equivalent PSID fields for backwards compatibility, though - sort of how MP3 ID3v1 and ID3v2 can coexist.) That would also help with STIL entries, too. I have no clue how Unicode chars would be displayed by PSID64, though. =)

I'm also in favor of adding UTF-8 support. PSID64 currently translates
ISO Latin 1 characters to PETSCII by removing any diacritics present
in these fields. Maybe for some Unicode characters a similar
translation is possible, others will probably just shown as '?'.

Regards,
Roland

--- CORRECTION in below example in SONGLENGTH

Hi all!

To get the ball rolling, here is a quick draft of a proposal I am putting forward: an extension of the current PSID/RSID format with new v3 fields added to it. The document is in Microsoft Word so that it can show you the changes I made to the PSID v2NG file format description, but I also included a plain-text version.

http://lala.c64.org/SID_file_format_v3.zip

Please note that all the string fields are UTF8 encoded. TITLE, AUTHOR, RELEASED and COMMENT (i.e. STIL INFO) can be repeated multiple times for multiple subsongs and/or for a file-global field.

My questions:

- Do we really need multiple RELEASED fields, or is just one per file sufficient? (Steppe, input please! :)

- I remember having a discussion about a new <speed> setting that allows for more than just 32 tunes, and new <clock>, <psidSpecific> and <sidModel> fields that could be set per subsong - do we still need these, or is it sufficient to have just one setting per SID file? (That is, are there SID files out there with subsongs that could be a mix of 6581 and 8580 tunes, etc.?)

To give you an example of how this file format would look like:

<up to 0x0007C offset everything looks as it used to in a regular PSID v2NG file, except that version number is 3 now and dataOffset = 0x0093, and let's say songs = 1>

FILE OFFSET - DATA
0x007C - [0x000C]    (total field length is 12)
0x007E - [0x00]      (TITLE field code)
0x007F - [0x0000]    (a file-global title)
0x0081 - ["Example"] (the title string itself in UTF8 encoding)
0x0088 - [0x0008]    (total field length is 8)
0x008A - [0x01]      (AUTHOR field code)
0x008B - [0x0000]    (a file-global author)
0x008D - ["Joe"]     (the author string itself in UTF8 encoding)
0x008E - [0x0005]    (total field length is 5)
0x0090 - [0x04]      (SONGLENGTH field code)
0x0091 - [0x0BB8]    (decimal 3000 = 300.0 seconds long)
0x0093 - <start of C64 data>

Clearly, this is a contrived example, because we could've used the legacy title and author fields due to the lack of special characters and the string length < 31 (and in fact, they should all be the same in this example), but you get the idea.
 
--

LaLa

Hi all!

To get the ball rolling, here is a quick draft of a proposal I am putting forward: an extension of the current PSID/RSID format with new v3 fields added to it. The document is in Microsoft Word so that it can show you the changes I made to the PSID v2NG file format description, but I also included a plain-text version.

http://lala.c64.org/SID_file_format_v3.zip

Please note that all the string fields are UTF8 encoded. TITLE, AUTHOR, RELEASED and COMMENT (i.e. STIL INFO) can be repeated multiple times for multiple subsongs and/or for a file-global field.

My questions:

- Do we really need multiple RELEASED fields, or is just one per file sufficient? (Steppe, input please! :)

- I remember having a discussion about a new <speed> setting that allows for more than just 32 tunes, and new <clock>, <psidSpecific> and <sidModel> fields that could be set per subsong - do we still need these, or is it sufficient to have just one setting per SID file? (That is, are there SID files out there with subsongs that could be a mix of 6581 and 8580 tunes, etc.?)

To give you an example of how this file format would look like:

<up to 0x0007C offset everything looks as it used to in a regular PSID v2NG file, except that version number is 3 now and dataOffset = 0x0094, and let's say songs = 1>

FILE OFFSET - DATA
0x007C - [0x000C]    (total field length is 12)
0x007E - [0x00]      (TITLE field code)
0x007F - [0x0000]    (a file-global title)
0x0081 - ["Example"] (the title string itself in UTF8 encoding)
0x0088 - [0x0008]    (total field length is 8)
0x008A - [0x01]      (AUTHOR field code)
0x008B - [0x0000]    (a file-global author)
0x008D - ["Joe"]     (the author string itself in UTF8 encoding)
0x008E - [0x0006]    (total field length is 6)
0x0090 - [0x0004]    (SONGLENGTH field code)
0x0092 - [0x0BB8]    (decimal 3000 = 300.0 seconds long)
0x0094 - <start of C64 data>

Clearly, this is a contrived example, because we could've used the legacy title and author fields due to the lack of special characters and the string length < 31, but you get the idea.
 
--

LaLa

> I was thinking exactly the same, LaLa. Flexible offset sounds cool. And probably 
> also the least hassle if people are converting player applications. And 
> incidentially this is just what Wilfred suggested a year or two ago when we 
> already had this discussion for a short while.

I do not know how rigorous other software developers were when implementing PSID loader routines, but technically, they should reject files if they a) don't start with either "PSID" or "RSID", b) have a version number larger than 2, c) have a dataOffset other than 0x0076 or 0x007C. I'm just thinking of backwards compatibility here.

For example, my XMPlay SID plugin crashes (!) on Windows when I change the version number in a SID file to 3. Ouch. I know that my own Audio::SID Perl module (available in CPAN) can deal with a dataOffset other than the above two values (the rest of the header will go into a non-publicized data field called "PADDING"), but it takes any version number greater than 1 as meaning "version 2" - so, that would have to change, too.

One reason we switched the magic ID to "RSID" was so that non-RSID-compatible players would reject those files. We could do the same (say, use "XSID" for version > 2).


BTW, add Unicode UTF-8 support to my list of desired features. So we could have longer and linguistically correct "title", "author" and "copyright" fields. (We could keep the equivalent PSID fields for backwards compatibility, though - sort of how MP3 ID3v1 and ID3v2 can coexist.) That would also help with STIL entries, too. I have no clue how Unicode chars would be displayed by PSID64, though. =)

--
LaLa

LaLa schrieb:
> To be honest, as I am looking at the current PSID v2NG/RSID file format spec, that can satisfy all of these requirements simply by bumping up the version number to 0x0003 and changing the dataOffset as needed. In other words, keep all the current fixed RSID fields, but in version 3 let the dataOffset be dynamic (per SID file) so the header can incorporate new variable-sized fields like STIL text, per tune info (e.g. songlengths) and the like. No need to re-invent the wheel, we just add to it. Whaddaya think?

I was thinking exactly the same, LaLa. Flexible offset sounds cool. And probably 
also the least hassle if people are converting player applications. And 
incidentially this is just what Wilfred suggested a year or two ago when we 
already had this discussion for a short while.

/Stephan

(Via the PSID64-devel list this time.)

> OK. I can only fully agree on this it's also not my intention to
> change that. On the contrary, one of the problems with some of the
> current SID files is the fact that multiple program files are merged
> into a single SID file in such a way that program data is overwritten,
> making it impossible to call the init function multiple times on a
> real C64.

I would encourage you to submit these as problem tunes to the HVSC Team for fixing. I'm sure they know how to handle them correctly.


As for the new SID file format itself, a couple of design goals should be kept in mind:

- It should be a compact format so that the files could be loaded into a C64 as is without much trouble.

- It should be easily extensible, so new fields can be added in the future without requiring a complete rewrite of the file format.

- It should have all the fields of PSID+RSID, plus as much information from STIL.txt, Songlength.txt, and maybe even BUGlist.txt as possible.


To be honest, as I am looking at the current PSID v2NG/RSID file format spec, that can satisfy all of these requirements simply by bumping up the version number to 0x0003 and changing the dataOffset as needed. In other words, keep all the current fixed RSID fields, but in version 3 let the dataOffset be dynamic (per SID file) so the header can incorporate new variable-sized fields like STIL text, per tune info (e.g. songlengths) and the like. No need to re-invent the wheel, we just add to it. Whaddaya think?
 
--

Imre Olajos