Commit 395a6c1f authored by cvs2svn's avatar cvs2svn
Browse files

This commit was manufactured by cvs2svn to create tag 'COOL_2_7_0'.

git-svn-id: file:///git/lcgcool.svndb/cool/tags/COOL_2_7_0@15592 4525493e-7705-40b1-a816-d608a930855b
parent c48d20a5
# $ Id: requirements,v 1.4 2005/08/24 17:31:07 marcocle Exp $
package RelationalCool
# Public dependencies
use CoolKernel v*
# Build rules
# Use lcg_plugin_library and add 'use Reflex' to make this a Reflex plugin
# Use lcg_module_library to make this a SEAL plugin (not linked by default)
apply_pattern lcg_shared_library
include_path none
# Link the CORAL relational libraries
macro_append lcg_RelationalCool_shlibflags ' $(Boost_linkopts_thread)'
macro_append lcg_RelationalCool_shlibflags '' \
WIN32 ' $(Boost_linkopts_date_time)'
macro_append lcg_RelationalCool_shlibflags '' \
WIN32 ' PowrProf.lib '
# Tests and utilities
use CppUnit v* LCG_Interfaces -no_auto_imports
# For the SEAL signal handler
###use SEAL v* LCG_Interfaces
# Setup the build options for the tests
# RelationalCool test linkopts
macro_append RelationalCool_testlinkopts " -llcg_RelationalCool " \
WIN32 " /libpath:../$(CMTCONFIG)/lib lcg_RelationalCool.lib "
macro_append RelationalCool_testlinkopts '' \
WIN32 ' $(Boost_linkopts_date_time)'
# RelationalCool utility linkopts
macro_append RelationalCool_utilitylinkopts " -llcg_RelationalCool " \
WIN32 " /libpath:../$(CMTCONFIG)/lib lcg_RelationalCool.lib"
macro_append RelationalCool_utilitylinkopts '' \
WIN32 ' $(Boost_linkopts_date_time)'
# Local patterns
pattern cool_no_db_test \
apply_pattern lcg_unit_test_application name=<package>_<name> \
files=../tests/<name>/*.cpp ; \
macro_append test_<package>_<name>_cppflags ' $(ppcmd)"../tests/Common"' ;\
macro_remove test_<package>_<name>linkopts '' Darwin '-L$(CMTINSTALLAREA)/$(CMTCONFIG)/tests/lib'
pattern cool_db_test \
apply_pattern lcg_unit_test_application name=<package>_<name> \
files=../tests/<name>/*.cpp ;\
macro_append test_<package>_<name>_cppflags ' $(ppcmd)"../tests/Common"' ;\
macro_append test_<package>_<name>linkopts ' $(Boost_linkopts_thread)' ;\
macro_remove test_<package>_<name>linkopts '' Darwin '-L$(CMTINSTALLAREA)/$(CMTCONFIG)/tests/lib'
pattern cool_utility \
application <name> ../utilities/<name>/*.cpp ;\
macro_append <name>linkopts ' $(RelationalCool_utilitylinkopts) $(Boost_linkopts_thread)' ;\
macro <name>_dependencies ' lcg_RelationalCool '
# The tests
apply_pattern cool_no_db_test name=ChannelSelection
apply_pattern cool_no_db_test name=PayloadSpecification
apply_pattern cool_no_db_test name=VersionNumber
apply_pattern cool_no_db_test name=HvsPathHandler
apply_pattern cool_no_db_test name=RelationalDatabaseId
apply_pattern cool_no_db_test name=ObjectId
apply_pattern cool_no_db_test name=utility_methods
apply_pattern cool_db_test name=Channels
apply_pattern cool_db_test name=HvsTags
apply_pattern cool_db_test name=RalDatabase
apply_pattern cool_db_test name=RalDatabase_extendedSpec
apply_pattern cool_db_test name=RalDatabaseSvc
apply_pattern cool_db_test name=RalSequence
apply_pattern cool_db_test name=RelationalFolder
apply_pattern cool_db_test name=RelationalFolderSet
apply_pattern cool_db_test name=RelationalObjectIterator
apply_pattern cool_db_test name=RelationalObjectMgr
apply_pattern cool_db_test name=RelationalObjectSet
apply_pattern cool_db_test name=RelationalObjectTable
# Private tests - not to be installed in the release area
###apply_pattern cool_db_test name=bug30858
###macro_remove tests_constituents test_RelationalCool_bug30858
###apply_pattern cool_db_test name=bug33578
###macro_remove tests_constituents test_RelationalCool_bug33578
# The utilities
apply_pattern cool_utility name=coolAuthentication
apply_pattern cool_utility name=coolDropDB
apply_pattern cool_utility name=coolDumpSchema
apply_pattern cool_utility name=coolEvolveSchema
apply_pattern cool_utility name=coolPrivileges
apply_pattern cool_utility name=coolReplicateDB
apply_pattern cool_utility name=coolStat
apply_pattern cool_utility name=coolValidateSchema
# Install the scripts
macro scrdir "$(CMTINSTALLAREA)/$(tag)/bin" \
WIN32 "$(CMTINSTALLAREA)\$(tag)\bin"
action install_scripts "mkdir -p $(scrdir); cp -r ../scripts/* $(scrdir)/." \
WIN32 "xcopy /I/S/Y/Q ..\scripts\*.* $(scrdir)\."
# Append to 'constituents' to execute an action in 'cmt make'
# (append to 'all_constituents' to execute it only in 'cmt make all').
# Remove the action from cmt_actions_constituents so that the action
# is not executed twice in 'cmt make all_groups' (it executes all actions).
macro_append constituents "install_scripts"
macro_remove cmt_actions_constituents "install_scripts"
# Fake target for examples
action examples "echo No examples in this package"
macro_remove cmt_actions_constituents "examples"
# Do not install the header files
macro_remove constituents 'install_includes'
2005.02.03 Schema evolution (AV, MC)
The new software does not allow to delete the old tables.
Long term solutions: provide schema evolution.
Short term solution: provide scripts to clean the schema.
AV's plan for cleaning up the code - 14.12.2004
General ideas:
- Need three or more types of classes, Relational, Ral, Hvs;
for the moment they can stay all in RelationalCool, then we can move them
- Hvs for the moment is only needed to model transient HVS entities
and factor out functionality that is implemented by CondDB stuff too
(eg HvsNode should be - again - a base clas for RelationalFolder)
- Relational classes should contain all the table and column names
and all the algorithmic logic of the relational implementation
- Ral classes should contain only the code that really does table
creation/dropping and row insertion/update/select/delete: the idea
is that one can provide a MySQL or OCCI direct implementation of
the Relational conddb by JUST replacing the Ral classes
==> Main emphasis of this round of cleanup should be to extract out of
the Ral classes the functionality that belongs to Relational
Specific points for Ral classes
- RalDatabaseSvc: OK should remain
- RalDatabaseSvcFactory: OK for now... eventually may be replaced
by a more general DatabaseSvcFactory, which should understand
IDs of the form "ral::mysql:/" or "mysql::mysql:/" and instantiate
a Ralor MySQL database service to handle the same mysql database....
- RalTransaction: should disappear, it should become RelationalTransaction.
The idea is that the user does not manipulate transactions.
All methoids called by the user are encapsulated within a transaction.
Typically, these methods have the form { 1.startTransaction
2.doSomethingInDb 3.commitTransaction }. Such methods should
go to the Relational classes. In particular, startTransaction and
commitTransaction should be achieved withRelationalTransaction, eg
RelationalTransaction transaction( m_db );
... interpret the row and return the object ...
RelationalTransaction( RelationalDatabase* db ) {
db->startTransaction( this );
The RalDatabase, ie the only Ral aware class, is the one that both
fetches the object and actually starts the transaction:
virtual RelationalDatabase startTransaction( RelationalTransaction t ) = 0;
virtual RelationalDatabase fetchRow(...) = 0;
RalDatabase::startTransaction( RelationalTransaction t ) {
t.setStarted(); // JUST SETS A FLAG...
RalDatabase::fetchRow(...) {
Forcing the startTransaction method to accept a RelationalTransaction
argument binds the startTransaction to a particular transaction instance,
so that one is not just tempted to call startTransaction instead
of writing "RelationalTransaction t" in the method where t is started.
Eventually, note that RDBMS offer the possibility to start transacions
with a name, so as to have several concurrent isolated transactions
in the same session... passing the argument can just pass the name...
- RalSequence: this should also change as RelationalSequence.
Again, RalDatabase should take care of doing the actual insert/delete
and so on in the database....
I would keep this for quite a while, because I am not sure when
RAL will support native sequences, or provide SYSDATE support.
Eventually one may think of having all sequences in the same table,
each sequence in a separate row, to have fewer tables.
- RalDatabase: this should be the ONLY Ral class (apart from the
databasesvc and its factory) that remains.
NB: for the moment, the methods it is answeriung are of the form
'fetch object from object table with that order'...
Eventually, this could be refactored slightly so that it offers
much higher level services, eg generic table handling, BUT
- we dont want to reinvent RAL
- somewhere the specific choices of how to write the queries in SQL
(eg the order of inserting the arguments) must be made,
and it should be in the RAL specific class
Other points:
- Should make much more uniform the treatment of the various tables
in the code, have a uniform look and feel for defining the column
names and column types. We may think of using typedefs for the
column types (in the static declarations), so that the code is not
forced to make assumptions in setValue/getValue (eg if IValidityKey
is declared as long long but the iovSince column is typedefed
temporarily as long, the code can be written without knowing
that the iovSince is long... it will just force a conversion
from IValidityKey to/from iovSince type, whatever that is)
- It may be (??) useful to have a table class, and derive all
tables that are present in our code from that class.
But not sure, what would this gain exactly?
Maybe we can let RalDatabase handle the base table, and then automatically
all derived tables by inheritance?
- Instead/in any case, it may be useful to split out the various
'namespaces' RelationalObjectTable, RelationalFolderTable and so on
as real classes RelationalObjectTable, RelationalFolderTable, and so on.
Definitely (again!), the look and feel should be more harmonious,
eg column definitions for RelationalObjectTable in file
RelationalObject.h, or RelationalObjectTable.h, not moving
from RalDatabase.h to RelationalDatabase.h and so on...
- Iterators
Eventually these should become real iterators which retrieve rows in bulk,
say 1000 by 1000, but not N millions rows all together if this is what is
in the table and in the iteration loop. For this reason it should
only have forward iteration. "Freezing" of the database
can be achived by only selecting rows whose insertion time,
or update time, is less than the time at which the iterator
was created....
The number of rows fetched in each roundtrip may be set by the user,
but it may also be an internal detail at the beginning.
That is to say: the initial implementation of the iterator
now returns an ObjectVectorIterator, but eventually
it should return something more complex!
- Update time
The feature outlined above needs an extra "update time" column
that should be changed when for instance the online columns
has +infinity transformed into a real time.
Note that also in the versioning mode you will have the need
for updating the rows (eg "HEAD unti objectId..." column).
// $Id: RAL_feedback.txt,v 1.17 2005-03-18 21:33:44 avalassi Exp $
* Altering table to add foreign key constraint should be possible a posteriori
The functionality needs to be added to IRelationalTableSchemaEditor
(or should be moved from IEditableTableDescription).
* Need option to switch off /tmp/pool.sql.log
This huge file is extremely useful for debugging.
But it may decrease performance. Is it possible to switch it off?
* POOL_2_0_1 does not translate correctly unsigned long for MyODBC
In POOL_2_0_0 I got the table created with UNSIGNED BIGINT.
In POOL_2_0_1 the type is missing and SQL fails.
* Tablespace definition via the API
This is presently only possible via the job options or environment
variables at the domain level. This should be possible ALSO at the table
level (and at the session level) through the API.
* Support for insert/select statements
Support 'insert into xxx select * from yyy'.
Needed by Sven for simpler tagging.
* MySQL plugin internal emulation of bulk retrieval
The MySQL performance should be better understood and optimized.
If there is a problem with missing bulk operations (so that the choice
is either to go one by one or retrieve ALL rows in a query), it may be
useful to improve the emulation by doing the following
- allow the users to specify a prefetch buffer, eg 10k
- issue a read only transaction
- issue the select statement using an order by clause (either the one
provided by the user, or one on the PK of the table, or throw an
exception if none exists and a prefetch buffer is specified)
- retrieve the first 10k rows
- retrieve the next 10k rows and so on
The read only transaction and order by clause are to be screened behind
the implementation and ensure that the fetching order is reproducible.
* MySQL API direct implementation
Priority low. May be a useful alternative to ODBC eventually.
* Switch to non-scrollable cursors in the MySQL plugin
Internal change: the default MySQL plugin impleentation should use
forward-only cursors. Although this is not guaranteed to cause a major
performance improvement, this should be tested as it had a major
impact for Oracle (an dthe MySQL implementation is still relatively slow).
* BLOB support in AttributeList and RAL
The user should be able to store/retrieve as a database BLOB a raw binary
memory buffer of a given size.
* Query output type definition
The defineOutput() method should not require that the size of the
AttributeListSpecification is the same as the one of the output list
as defined with the successive calls to the addToOutput() method.
It should be enough to specify via defineOutput an AttributeListSpecification
that is a superset (ie larger) than all columns queried.
* MySQL plugin messaging
In DEBUG mode it should print out the statement that was prepared
in prepareStatement(), just like for Oracle.
* MySQL plugin support for Windows.
Presently it is only available under Linux.
* NULL value insertion.
Inserting NULL values should be done explicitly.
The best solution would be to modify AttributeList so that each
attribute in the list has the optionof being NULL/undefined. This
could then be used transaparently in RAL for both insertion and retrieval.
* String size specification
The user should be able to specify (and retrieve in the table description)
for a string column its maximum size and whether the latter is variable
or fixed.
Additionally, RAL should automatically store strings as CLOBs if they are
larger than the maximum VARCHAR size.
* MySQL plugin handling of multiply-bound variables
The user should be able to use the same syntax "where xx < :a and :a < yy"
as for Oracle, with a bind variable appearing twice. RAL should take care
of binding the variable twice.
* MySQL plugin handling of bind variable names
The user should be able to use the same syntax "where :a<yy" as for Oracle.
Presently this does not work because this is replaced by "?", interpreting
this as the binding of a variable called "a<yy" (ie presently one has to
write ":a <yy" instead of ":a<yy", taking care to leave a space).
RAL should interpret as the bind variable name only letters, numbers and
the "_" character. Any character other than these (eg +,-,<,>,=) should
indicate that the bind variable name is over.
* MySQL plugin handling of bind variable order
Presently the MySQL plugin assumes that the order in which attributes
are specified in the AttrinbuteList for a WHERE clause is the same as
the order in which the variables are bound. This should be changed
as it leads to errors that are difficult to trace and debug and
may go unnoticed. The MYSQL plugin should behave like Oracle OCI
and crosscheck names of variables (this would also solve the problem
of doubly dound variable names).
The user should be able to enable SQL_TRACE in a session.
For Oracle, this should issue
alter session set events '10046 trace name context forever, level 12'
Ideally this should be a session property (method of IRelationalSession).
Initially, this can be a domain property, set via the seal opts or
one of the 'hidden' environment variables, eg POOL_ORA_SQL_TRACE.
* Support for SYSDATE date computed by the server
1. The user should be able to retrieve into C++ the date on the server.
There should be a simple method to retrieve only the date. This would issue:
Oracle (systimestamp has higher precision - microseconds instead of seconds)
select sysdate from dual;
select systimestamp from dual;
select current_timestamp from dual;
select current_timestamp;
The ANSI version current_timestamp works for both but gives different results.
For instance, add a method serverCurrentTimestamp() in the IRelationalSession?
Priority: high.
2. The user should be able to retrieve the date when retrieving data
from any table. This would issue for instance
select [user columns], current_timestamp from [user table];
select [user columns], current_timestamp from [user table];
For instance, define a reserved keyword CURRENT_TIMESTAMP which can be used
in the IRelationalQuery::addToOutputList() method, without executing the
cross-check whether this exists in the table because it does not?
Priority: lower (if you can at least get the serverCurrentTimestamp
into C++, you can always do this in two steps).
3. The user should be able to insert the date into a column of a table.
This would issue for instance
insert [user values], current_timestamp into ...;
Priority: lower (if you can at least get the serverCurrentTimestamp
into C++, you can always do this in two steps).
PS: actually the priority may be a bit higher.
We need to COMPARE date values, presently we are comparing strings...
Note: presently COOL is working around this problem by using the UPDATE
clause, the only place where free SQL can be used: for instance,
- insert a dummy value into a table
- update the table with the system date
- read it back from the table to know what is the time on the server
* Support for seal::Time and native Oracle DATE
The user should be able to create table with DATE columns.
If possible, seal::Time should also be supported in
AttributeList and mapped to DATE SQL types by default.
The user should be able to analyze a table from the C++.
Although this is an administration level operation, it is essential
for fast tests where one wants to create a table, insert data and
select from the table at the same time. The table must be analyzed
after data have been inserted and before issuing the select query.
For instance, add a method analyzeTable to IRelationalTable?
This would issue
analyze table xxx compute statistics (Oracle)
analyze table (MySQL)
* ORA-01466
Oracle readonly transaction uses READ ONLY rather than SERIALIZABLE.
If a table is created and immediately queried, this causes an ORA-01466 error.
This was reported to Oracle, but their answer is that this is a feature
that needs to be better documented, not a bug that will be fixed,
and the workaround is to wait for a few seconds.
Given this situation, I would suggest the following workaround to apply
within RAL, which would at least prevent the problem from happening within
the same active session: an active OracleSession instance should keep
track of the last date at which it issued any kind of DDL statements.
Whenever a read-only transaction is issued, RAL itself should check
whether at least N seconds have elapsed since the last DDL: if not,
* Read only transactions (more generally)
We should review whether Oracle READ ONLY transactions are really
what we need for "read only" mode. SERIALIZABLE transactions look
safer and better than readonly, but they may block
concurrent users from running at the same time.
* AttributeList copy constructor and assignment operator
AttributeLists behave differently depending on whether they were built
from a reference or a boost shared pointer specification.
The assignment operator is reliable only in the second case.
The copy constructor gives problems for both.
This was notified to Kuba, Ioannis and Rado.
* Multi-threading problems with MySQL/ODBC
I had to disable the -pthread switch on RelationalCool because
the application hangs (only for MySQL). The circumstances are
difficult to explain exactly: it happens in RalSequence when
I try to drop a table and then recreate it again. It does not
happen if the table had been dropped using TOra.
The problem had already been mentioned to Rado long ago.
Has this been solved in the meantime (by RAL or by the next ODBC driver)?
[In COOL a workaround is presently used in RelationalCoool/BuildFile]
* Problem inserting rows where all values are NULL
Bug: when inserting rows where all values are null, this is translated
into non-valid SQL:
Maybe this has already been fixed?
* Partitioning and local indexes
The user should be able to specify that a table should be created
as partitioned and indexese are local. Priority: low.
* Support for 64-bit integer
This type could not be defined in an AttributeList or stored using RAL.
* Set clause syntax for MySQL/ODBC in IRelationalTableDataEditor::updateRows
The Oracle plugin required ":bindvariable", MySQL/ODBC required "?".
Now both use ":bindvariable".
* Feature with columnNamesAndTypes()
If you insert a column via insertColumn( columnName, typeName )
and then retrieve columnNamesAndTypes(), the types are not guaranteed
to be the same. For instance, insertColumn for "unsigned long" will
return an "unsigned long" on MySQL (as the native type can be mapped),
while it will return "long" on Oracle because this maps to NUMBER(38).
This causes problems if you try to then insert an attribute value via
setValue<type> into the attribute list, pool::attribute_bad_type
will be thrown.
=> Solution: when filling in the table, do not rely on columnNamesAndTypes():
instead, keep a local copy of the C++ mapping in the tables or in the
C++ code! This solves the setValue() problem.
* Feature with IRelationalCursor::currentRow()
Also in this case the same problem appears: the attribute list returned
has a specification that is read from the table description, ie exactly
the same returned by columnNamesAndTypes(). If the previous problem
appeared for setValue(), this appears for getValue(): it is a priory
impossible to know which C++ should be used for getValue(). The only
safe one is string, but this is not satisfactory!
=> Solution: you must use defineOutput() to specify the required format!
* NULL values inserted in MySQL third column (POOL_1_8_0)
This is fixed in the HEAD of ODBCAccess since december 3.
There was a bug with the use of STL vectors: push_back changes
the internal memory addresses of all vector elements.
* MyODBC bulk inserter inserts \0-terminated strings
Check by selecting
select s, length(s) from COOLTEST_F0001_IOVS;
These two queries gave different results (and only the second works fine!):
select * from COOLTEST_F0001_IOVS where s='Object 1';
select * from COOLTEST_F0001_IOVS where s='Object 1\0';
Fixed in POOL_2_0_2.
Various problems with CLOB. Let's start with the most important.
Problem #1: segmentation fault in reading back CLOB from Oracle (08.04.2005)
The problem initially appeared in COOL after changing the folder payload spec