Adding instruments to DB silently fails

[robertaramar]

Hi,
I am on 2.0.0.svn25 (Build 3002). Regardless which tool I am using, I cannot add instruments to the database.
I have created the DB and I see it being created.
Whenever a ADD DB_INSTRUMENTS .... is issued I get an OK, however, I cannot see the newly created instrument.
I have tried directories, files, SFZ, SF2, GIG all to no avail.
Any clues?
Regards,
Robert

[robertaramar]

Here is a debug output and the DB information that doesn't really get me any further.

Code: Select all

~/src/linuxsampler/linuxsampler/src$ ./linuxsampler --instruments-db-location /home/rschneid/.jsampler/instruments.db
Condition:: constructor, bInitialCondition=0
Condition:: constructor, bInitialCondition=0
Condition:: constructor, bInitialCondition=0
Condition:: constructor, bInitialCondition=0
LinuxSampler 2.0.0.svn25
Copyright (C) 2003,2004 by Benno Senoner and Christian Schoenebeck
Copyright (C) 2005-2016 Christian Schoenebeck
Binary built: Oct 10 2016
Detected features: MMX SSE SSE2
Automatic Stacktrace: Off
Creating Sampler...OK
Registered sampler engines: 'GIG','SF2','SFZ'
Registered MIDI input drivers: ALSA,JACK
Registered audio output drivers: ALSA,JACK
Loading instrument editor plugins...Could not open instrument editor plugins directory ('/usr/local/lib/linuxsampler/plugins'): No such file or directory
Registered instrument editors: 
Registered internal effect systems: LADSPA
no more csLADSPA plugins
failed to load DLL: '/usr/lib/ladspa/sifter_1210.so', cause: /usr/lib/ladspa/sifter_1210.so: undefined symbol: q_sort
Registered internal effects: 333
Starting LSCP network server (0.0.0.0:8888)...Condition:: constructor, bInitialCondition=0
Condition:: constructor, bInitialCondition=0
Condition::WaitIf: bCondition=0  TimeoutSeconds=0  TimeoutNanoSeconds=0
Condition::Waitif() -> LOCK()
Condition::Waitif() -> LOCK() passed
Condition::Waitif() -> waiting for 'true' condition
Condition::Set() -> LOCK()
Condition::Set() -> LOCK() passed
Condition::Set() -> broadcasting 'true' condition
Condition::Set() -> LOCK()
Condition::Set() -> LOCK() passed
Condition::Set() -> broadcasting 'true' condition
Condition::Waitif() -> awakened from 'true' condition waiting
Condition::WaitIf: bCondition=0  TimeoutSeconds=0  TimeoutNanoSeconds=0
Condition::Waitif() -> LOCK()
Condition::Waitif() -> LOCK() passed
Condition::WaitAndUnlockIf() -> UNLOCK()
Condition::WaitAndUnlockIf() -> UNLOCK() passed
OK
LinuxSampler initialization completed. :-)


LSCPServer: Client connection established on socket:8.
LSCPServer: Got command on socket 8, calling parser.
LSCPServer: [SET SHELL DOC 1
]
LSCPServer: SetShellDoc(val=1.000000)
LSCPServer::AnswerClient(ReturnMessage='OK
')LSCPServer: Done parsing on socket -1.
LSCPServer: Got command on socket 8, calling parser.
LSCPServer: [SET SHELL AUTO_CORRECT 1
]
LSCPServer: SetShellAutoCorrect(val=1.000000)
LSCPServer::AnswerClient(ReturnMessage='OK
')LSCPServer: Done parsing on socket -1.
LSCPServer: Got command on socket 8, calling parser.
LSCPServer: [SET SHELL INTERACT 1
]
LSCPServer: SetShellInteract(val=1.000000)
LSCPServer::AnswerClient(ReturnMessage='OK
')LSCPServer: Done parsing on socket -1.
LSCPServer::AnswerClient(ReturnMessage='SHU:2:A{{CU}}{{GF}}{{PB}}ADD | APPEND
')LSCP doc reply -> ''
LSCPServer::AnswerClient(ReturnMessage='SHU:0:ADD DB_INSTRUMENTS '/' '/home/rschneid/Documents/Music/soundfonts/flstudiomusic.com/Bass/1276-FReTLeZzz.sf2' 0{{CU}}{{GF}}{{PB}}<number>
')LSCP doc reply -> 'SHD:1:ADD DB_INSTRUMENTS
The front-end can add one or more instruments to the instruments database by sending the following command:

ADD DB_INSTRUMENTS [NON_MODAL] [<mode>[ FILE_AS_DIR]] <db_dir> <file_path> [<instr_index>]

Where <db_dir> is the absolute path name of a directory (encapsulated intoapostrophes) in the instruments database in which only the new instruments (that are not already in the database) will be added, <file_path> is the absolute path name of a file or directory in the file system (encapsulated into apostrophes). In case an instrument file is supplied, only the instruments in the specified file will be added to the instruments database. If the optional <instr_index> (the index of the instrument within the given file) is supplied too, then only the specified instrument will be added. In case a directory is supplied, the instruments in that directory will be added. The OPTIONAL <mode> argument is only applied when a directory is provided as <file_path> and specifies how the scanning will be done and has exactly the following possibilities:

"RECURSIVE" - All instruments will be processed, including those in the subdirectories, and the respective subdirectory tree structure will be recreated in the instruments database

"NON_RECURSIVE" - Only the instruments in the specified directory will be added, the instruments in the subdirectories will not be processed.

"FLAT" - All instruments will be processed, including those in the subdirectories, but the respective subdirectory structure will not be recreated in the instruments database. All instruments will be added directlyin the specified database directory.

If FILE_AS_DIR argument is supplied, all instruments in an instrument file will be added to a separate directory in the instruments database, which name will be the nameof the instrument file with the file extension stripped off.

The difference between regular and NON_MODAL versions of the command is that the regular command returns when the scanning is finished while NON_MODAL version returns immediately and a background process is launched. The GET DB_INSTRUMENTS_JOB INFO command can be used to monitor the scanning progress.

Possible Answers:

"OK" - on success when NON_MODAL is not supplied

"OK[<job-id>]" - on success when NON_MODALis supplied, where <job-id> is a numerical ID used to obtain status information about the job progress. See GET DB_INSTRUMENTS_JOB INFO 

"ERR:<error-code>:<error-message>" - if an invalid path is specified.

Examples:

C: "ADD DB_INSTRUMENTS '/Piano Collection' '/home/me/gigs/PMI Bosendorfer 290.gig' 0"

S: "OK"


.'
LSCPServer::AnswerClient(ReturnMessage='SHD:1:ADD DB_INSTRUMENTS
The front-end can add one or more instruments to the instruments database by sending the following command:

ADD DB_INSTRUMENTS [NON_MODAL] [<mode>[ FILE_AS_DIR]] <db_dir> <file_path> [<instr_index>]

Where <db_dir> is the absolute path name of a directory (encapsulated intoapostrophes) in the instruments database in which only the new instruments (that are not already in the database) will be added, <file_path> is the absolute path name of a file or directory in the file system (encapsulated into apostrophes). In case an instrument file is supplied, only the instruments in the specified file will be added to the instruments database. If the optional <instr_index> (the index of the instrument within the given file) is supplied too, then only the specified instrument will be added. In case a directory is supplied, the instruments in that directory will be added. The OPTIONAL <mode> argument is only applied when a directory is provided as <file_path> and specifies how the scanning will be done and has exactly the following possibilities:

"RECURSIVE" - All instruments will be processed, including those in the subdirectories, and the respective subdirectory tree structure will be recreated in the instruments database

"NON_RECURSIVE" - Only the instruments in the specified directory will be added, the instruments in the subdirectories will not be processed.

"FLAT" - All instruments will be processed, including those in the subdirectories, but the respective subdirectory structure will not be recreated in the instruments database. All instruments will be added directlyin the specified database directory.

If FILE_AS_DIR argument is supplied, all instruments in an instrument file will be added to a separate directory in the instruments database, which name will be the nameof the instrument file with the file extension stripped off.

The difference between regular and NON_MODAL versions of the command is that the regular command returns when the scanning is finished while NON_MODAL version returns immediately and a background process is launched. The GET DB_INSTRUMENTS_JOB INFO command can be used to monitor the scanning progress.

Possible Answers:

"OK" - on success when NON_MODAL is not supplied

"OK[<job-id>]" - on success when NON_MODALis supplied, where <job-id> is a numerical ID used to obtain status information about the job progress. See GET DB_INSTRUMENTS_JOB INFO 

"ERR:<error-code>:<error-message>" - if an invalid path is specified.

Examples:

C: "ADD DB_INSTRUMENTS '/Piano Collection' '/home/me/gigs/PMI Bosendorfer 290.gig' 0"

S: "OK"


.
')LSCP Shell <- expect mult-line response
LSCPServer::AnswerClient(ReturnMessage='SHU:0:ADD DB_INSTRUMENTS '/' '/home/rschneid/Documents/Music/soundfonts/flstudiomusic.com/Bass/1276-FReTLeZzz.sf2' 0{{CU}}{{GF}}
')LSCP doc reply -> ''
LSCPServer: Got command on socket 8, calling parser.
LSCPServer: [ADD DB_INSTRUMENTS '/' '/home/rschneid/Documents/Music/soundfonts/flstudiomusic.com/Bass/1276-FReTLeZzz.sf2' 0
]
LSCPServer: AddDbInstruments(DbDir=/,FilePath=/home/rschneid/Documents/Music/soundfonts/flstudiomusic.com/Bass/1276-FReTLeZzz.sf2,Index=0,bBackground=0)
InstrumentsDb: AddInstruments(DbDir=/,FilePath=/home/rschneid/Documents/Music/soundfonts/flstudiomusic.com/Bass/1276-FReTLeZzz.sf2,Index=0,bBackground=0)
InstrumentsDb: AddInstruments(DbDir=/,insDir=0,FilePath=/home/rschneid/Documents/Music/soundfonts/flstudiomusic.com/Bass/1276-FReTLeZzz.sf2,Index=0)
InstrumentsDb: GetDirectoryId(Dir=/)
InstrumentsDb: AddInstrumentsFromFile(DbDir=/,File=/home/rschneid/Documents/Music/soundfonts/flstudiomusic.com/Bass/1276-FReTLeZzz.sf2,Index=0)
LSCPServer::AnswerClient(ReturnMessage='OK
')LSCPServer: Done parsing on socket -1.
LSCPServer::AnswerClient(ReturnMessage='SHU:0:{{CU}}{{GF}}
')LSCP doc reply -> 'SHD:0'
LSCPServer::AnswerClient(ReturnMessage='SHD:0
')LSCPServer: Got command on socket 8, calling parser.
LSCPServer: [
]
LSCPServer: Done parsing on socket -1.
LSCPServer::AnswerClient(ReturnMessage='SHU:2:L{{CU}}{{GF}}{{PB}}LIST | LOAD
')LSCP doc reply -> ''
LSCPServer::AnswerClient(ReturnMessage='SHU:0:LIST DB_INSTRUMENTS RECURSIVE '/'{{CU}}{{GF}}{{PB}}<path>
')LSCP doc reply -> 'SHD:1:LIST DB_INSTRUMENTS
The front-end can retrieve the current list of instruments in specific directory by sending the following command:

LIST DB_INSTRUMENTS [RECURSIVE] <dir>

Where <dir> should be replaced by the absolute path name of the directory. If RECURSIVE is specified, the absolute path names of all instruments, including those located in subdirectories ofthe specified directory, will be returned.

Possible Answers:

A comma separated list of all instruments (encapsulated into apostrophes) in the specified directory.

"ERR:<error-code>:<error-message>" - if the given directory does not exist.

Example:

C: "LIST DB_INSTRUMENTS '/Piano Collection'"

S: "'Bosendorfer 290','Steinway D'"

C: "LIST DB_INSTRUMENTS RECURSIVE '/Piano Collection'"

S: "'/Piano Collection/Bosendorfer 290','/Piano Collection/Steinway D','/Piano Collection/Lite/Free Piano'"


.'
LSCPServer::AnswerClient(ReturnMessage='SHD:1:LIST DB_INSTRUMENTS
The front-end can retrieve the current list of instruments in specific directory by sending the following command:

LIST DB_INSTRUMENTS [RECURSIVE] <dir>

Where <dir> should be replaced by the absolute path name of the directory. If RECURSIVE is specified, the absolute path names of all instruments, including those located in subdirectories ofthe specified directory, will be returned.

Possible Answers:

A comma separated list of all instruments (encapsulated into apostrophes) in the specified directory.

"ERR:<error-code>:<error-message>" - if the given directory does not exist.

Example:

C: "LIST DB_INSTRUMENTS '/Piano Collection'"

S: "'Bosendorfer 290','Steinway D'"

C: "LIST DB_INSTRUMENTS RECURSIVE '/Piano Collection'"

S: "'/Piano Collection/Bosendorfer 290','/Piano Collection/Steinway D','/Piano Collection/Lite/Free Piano'"


.
')LSCP Shell <- expect mult-line response
LSCPServer::AnswerClient(ReturnMessage='SHU:0:LIST DB_INSTRUMENTS RECURSIVE '/'{{CU}}{{GF}}
')LSCP doc reply -> ''
LSCPServer: Got command on socket 8, calling parser.
LSCPServer: [LIST DB_INSTRUMENTS RECURSIVE '/'
]
LSCPServer: GetDbInstruments(Dir=/,Recursive=1)
InstrumentsDb: GetInstruments(Dir=/,Recursive=1)
InstrumentsDb: BeginTransaction(InTransaction=0)
InstrumentsDb: GetDirectoryId(Dir=/)
InstrumentsDb: ExecSqlInt(Sql=SELECT parent_dir_id FROM instr_dirs WHERE dir_id=0)
InstrumentsDb: GetDirectoryId(Dir=/)
InstrumentsDb: ExecSqlStringList(Sql=SELECT dir_name FROM instr_dirs WHERE parent_dir_id=0 AND dir_id!=0)
InstrumentsDb: EndTransaction(InTransaction=0)
LSCPServer::AnswerClient(ReturnMessage='
')LSCPServer: Done parsing on socket -1.

Code: Select all

sqlite> .schema
CREATE TABLE instr_dirs (                                            dir_id         INTEGER PRIMARY KEY AUTOINCREMENT,                parent_dir_id  INTEGER DEFAULT 0,                                dir_name       TEXT,                                             created        TIMESTAMP DEFAULT CURRENT_TIMESTAMP,              modified       TIMESTAMP DEFAULT CURRENT_TIMESTAMP,              description    TEXT,                                             FOREIGN KEY(parent_dir_id) REFERENCES instr_dirs(dir_id),        UNIQUE (parent_dir_id,dir_name)                              );
CREATE TABLE instruments (                                      instr_id        INTEGER PRIMARY KEY AUTOINCREMENT,          dir_id          INTEGER DEFAULT 0,                          instr_name      TEXT,                                       instr_file      TEXT,                                       instr_nr        INTEGER,                                    format_family   TEXT,                                       format_version  TEXT,                                       instr_size      INTEGER,                                    created         TIMESTAMP DEFAULT CURRENT_TIMESTAMP,        modified        TIMESTAMP DEFAULT CURRENT_TIMESTAMP,        description     TEXT,                                       is_drum         INTEGER(1),                                 product         TEXT,                                       artists         TEXT,                                       keywords        TEXT,                                       FOREIGN KEY(dir_id) REFERENCES instr_dirs(dir_id),          UNIQUE (dir_id,instr_name)                              );
sqlite> select * from instr_dirs;
0|-2|/|2016-10-09 15:42:55|2016-10-09 15:42:55|
sqlite> select * from instruments;
sqlite> 

[cuse]

The instruments DB feature maintenance is orphaned for quite a while now. Grigor used to work on it, so it hasn't seen an update in several years now.

Looking at your debug output though, I see what the problem is: you are adding sound font files, and when the current instruments DB feature saw its last (major) updates, support for SF2 and SFZ by LinuxSampler did not even exist yet!

So right now the current instruments DB implementation just looks for .gig files when scanning for instruments. When I find some time, I can look for adding the missing code for SF2 and SFZ files. In the meantime, in case you want to help adding support for this, have a look at src/db/InstrumentDB.cpp. At line 1153 (method AddInstrumentsFromFile()) it checks the file name extension (i.e. ".gig"), and method AddGigInstruments() adds the respective .gig file to the DB. So the missing code for SF2 and SFZ must be added there. If you need some help on this, just let me know. You may also consider to subscribe on the linuxsampler-devel mailing list in that case, since the LinuxSampler developers are not reading the forums every day.

[robertaramar]

Hi cuse,

thanks for the answer. I guess I will not subscribe to the mailing list. I am really, really old, but mailing-lists are still older Isn't there any other way to communicate with the devs?

I have gdb'ed adding a GIG file but still, I have trouble when I add the "Philharmonia Trumpet long loud" from Open Orchestra, which remains in my folder '/home/rschneid/Documents/Music/soundfonts/linuxsampler/philharmonia_trumpet_long_loud.gig', this is what I get in the DB:

Code: Select all

1|0| |��^��|0|GIG|3|5275716|2016-10-15 15:28:57|2016-10-15 15:28:57||0|0|Philharmonia Trumpet long lou|
2|0| |0�^��|0|GIG|3|5275716|2016-10-15 15:44:54|2016-10-15 15:44:54||0|0|Philharmonia Trumpet long lou|

During stepping through the methods in InstrumentsDb.cpp, I couldn't see any issue, however, I wasn't able to see the pstmt content. I am not so familiar with using gdb.

I have actually added the same file twice, and see, the BLOB content of the instr_file is different. Are we facing a memory overwrite here?

Regards,

Robert

[robertaramar]

I have gone one step further and as I said in my previous post, the Strings for path, name, artist etc. are correctly passed into BindTextParam, however they are corrupted in the database.
I would assume, the strings are somehow on the heap which is cleared up before the sql-engine writes the data to the database file.

I wanted to file a bug, but I do not have a bugzilla account and I cannot create one as Account Creation is disabled and I do not know how to read Andreas Persson.

I would really appreciate if someone could look into that.

[robertaramar]

And one more finding. The memory corruption doesn't seem to be create only, it also seems to occur on reads to the DB.
I have manually filled the DB with entries and at some, I can get a GET DB_INSTRUMENT INFO and on some I can't. There is nothing in the records that would explain that.
BindTextParam seems to have severe issue.

I don't know whether this is the proper fix, but it cures my problem:

InstrumentsDb.cpp#1654:
int res = sqlite3_bind_text(pStmt, Index, Text.c_str(), -1, SQLITE_TRANSIENT);

[cuse]

thanks for the answer. I guess I will not subscribe to the mailing list. I am really, really old, but mailing-lists are still older Isn't there any other way to communicate with the devs?

You mean like a LinuxSampler WhatsApp group? No, the only reliable and fastest way to discuss developer issues is the linuxsampler-devel maling list. It is simply the least common denominator, because not everybody uses Google+, FB, etc. , but everybody got an email address. I would agree with you if it was an 80s style newsgroup (where you had to use special clients etc.), but in case of mailing lists I don't really see a disadvantage.

I don't know whether this is the proper fix, but it cures my problem:

InstrumentsDb.cpp#1654:
int res = sqlite3_bind_text(pStmt, Index, Text.c_str(), -1, SQLITE_TRANSIENT);

That fix is correct. SQLITE_TRANSIENT causes the supplied Text to be copied immediately, which was not the case with SQLITE_STATIC, and hence this was a severe bug causing undefined behavior of libsqlite, since "Text" is just a temporary variable.

I am not used to the instruments DB code, but there may still be further bugs like this one. For example sqlite3_prepare() should probably be replaced by sqlite3_prepare_v3(). The SQLite docs are not clear on that, but it could be that the old sqlite3_prepare() function is not copying the original SQL statement text, which then might also cause undefined behavior due to the same cause as above.

[cuse]

Just minor update on this one: that instruments DB issue should be fixed in the meantime in SVN.