LISTSERV mailing list manager LISTSERV 16.0

Help for ZNG Archives

ZNG Archives

ZNG Archives

[email protected]












By Topic:










By Author:











Monospaced Font



ZNG Home

ZNG Home

ZNG  December 2004

ZNG December 2004


Re: Adlib Base profile


Hedzer Westra <[log in to unmask]>


Z39.50 Next-Generation Initiative


Fri, 17 Dec 2004 15:47:40 +0100





text/plain (310 lines) , AdlibBaseProfile.txt (1 lines)

Hi all,

I received some 'complaints' and suggestions about the Adlib Base
Profile from Mike & Rob. Here are my replies. I would like the Base
Profile to be final before I start reprogramming, so here are my answers
to their remarks. The new Base Profile (Beta-2) has been attached.

Many thanks for your help so far!

>> - The meta-index cql.anywhere searches all indexes defined in the
>> Adlib database at once. It does not search all indexes in all
>> context sets, as the CQL context set suggests.
>It is at best inadvisable, and probably just wrong, to _re_define the
meaning of an existing index like
>this -- especially such a core one. If cql.anywhere doesn't meet your
needs, it would be better to define
>your own index that does (or ask to have it added to the CQL set if you
think it's of general interest).
>I think that's exactly what cql.anywhere means. Search all indexes
that you know about, but you can't be
>expected to search indexes that you don't know about.
>Which is (thankfully, for once) exactly how it's defined:
>This means "search all indexes from all context sets you know".
Okay, okay, you're right. I was just being lazy; iterating *all* indexes
is easier than iterating over all
CQL-linked indexes. I'll create an adlib.allIndexes index for the
current behaviour, and program cql.anywhere
like it should be.

>> - The adlib.record meta-index searches the whole record. The operator
doesn't matter.
>Perhaps we should consider adding cql.record for whole-record searching
(where supported).
You can if you wish. I'll implement it for 1.2 if it's added then.

>You can't really say "the operator doesn't matter" as this is
overriding established semantics of CQL and
>the CQL context set. It would be much better to say "the operator must
be '=': all others will be rejected".
Will do, good suggestion.

>I'm not sure what exactly happens here. Is it:
>(a) You send some terms and it searches the entire record. (And how
is this different to cql.anywhere in
> practice?)
Not all fields are indexed.

>(b) You send the entire record as a term, and it returns the entire
record back to you if it finds it.
>The former. No-one wants (b).

> it is conventional in CQL to talk of "relations" rather then
"operators". Unless you have
> a compelling reason, you should probably stick to this convention.
I'll change that.

>> - The Adlib thesaurus operators 'adlib.generic', 'adlib.broader',
>> 'adlib.narrower', 'adlib.related', 'adlib.topterm' and
>> 'adlib.parents' do thesaurus-enabled searches. These only work
>> correctly on indexes with thesaurus links defined. Otherwise, they
fall back on '=' searching.
> Are these relations or relation modifiers? If you don't already have
this nailed down, I would recommend
> the latter, as they are all refinements on the general relation of
Hmm, good suggestion. They're relations now, because 1. they're
operators in the Adlib Expert Search
language and 2. I was too lazy to implement modifier handling :-)

I'll change it to: 'term =/adlib.thesaurus=generic myTerm' etc. if that
suits everyone.

>There really should be a thesaurus-use context set defined outside of
Adlib, for use in this and other
>profiles (or the relevant elements should be added to the existing
Zthes context set).
generic, topterm and parents are rather esoteric operators so I guess
bt, nt and rt are just fine.
Adlib does use/uf substitution automatically, and has no concept of le,
so those three aren't supported as
operators. (FYI I'm using 'operator' here because I'm referring to Adlib
expert search.)

>We actually started this process a month or two back, but got
sidetracks -- or maybe mired in excess
Are there any concrete results from that discussion, or was it just
that: a discussion?

>Depending on the urgency of your Adlib work, you might try to restart
that process and use the resulting
>"official" thesaurus-expansion support.
I just heard yesterday from our customer that Q1 2005 will be a testing
phase, and Q2/Q3 will be the
implementation phase so there's enough time to reach an agreement. To
top that, the customer probably won't
need explicit thesaurus support, I just added it to offer to SRU/CQL as
much functionality as the Adlib
query language has.

>> - The 'encloses' and 'within' operators are implemented using the
>> Adlib WHEN operator. Some examples:
>> 'term encloses "2000 2004"' translates to 'term >= 2000 WHEN term
<= 2004'
>Nope -- "encloses" is the converse of "within", so
> term encloses "2000 2004"
>translates to
> term <= 2000 WHEN term >= 2004
Erhm that would always be the empty set.. WHEN is just like AND, only a
bit different. Read the Adlib manual for details :-)

>> 'term within "2001 2005"' translates to 'term > 2001 WHEN term <
>My reading of the CQL context set indicates that this relation is
inclusive of endpoints, so you should
>translate to term >= 2001 WHEN term <= 2005
> within "2000 2004"
> -> Does the record contain a date between (inclusive) 2000 and 2004
I'll add a modifier: within/adlib.range=exclusive (or inclusive -
default, rightexclusive, leftexclusive).
Makes everybody happy, right? ;-)

> foo.rangeOfDates encloses 2002
> -> Does 2002 fall within (inclusive) the date range in the record.
> <record>
> <dateRange>2000 2004</dateRange>
> </record>
> would match the encloses query.
> (Hopefully this is how your internal queries work)
No they don't. I'm a bit confused now about encloses works. In the
manual they're not really well defined,
if I may say so. If encloses specifically needs a two-dimensional index
and a single search term, Adlib
can't support it. But Mike's example (the one which results in the empty
set) seems to suggest otherwise..

>> - there are two types of modifiers: data type modifiers and pattern
>> modifiers [...]
>This whole section belongs in the CQL context-set document.
I know, but since it isn't there I thought I'd just make it explicit in
Adlib's profile, so that it's at
least defined somewhere..

>> The pattern modifiers are:
>> cql.masked
>> cql.unmasked (not defined in CQL context set)
>We should fix that!
Something for a 1.2 proposal?

>> Note that the CQL context set is not required by the SRW Base
>That's not really true, as the CQL context set provides some of the key
elements used in pretty much CQL
>queries, e.g. the meaning of all the relations. Probably the base
profile should make this explicit.
Fine by me, I just wanted to make clear that Adlib's implementation
violates some of the CQL context set
descriptions. Pending this discussion, I'll clear up all or most of
these violations.

>> The modifiers cql.word and cql.string can not relate directly to
>> Adlib term or word matching because this is defined per index by
>> the user; in Adlib each index can be either word or term
>> indexed. If required, a field can be indexed by term as well as by
>[I don't understand this fully, I think because it assumes you know
something about Adlib. Is Adlib's
>"term" searching similar to what we mean by "string"?]
Yes, it is. I'll try to use CQL terminology in the next version of the

>> These two indexes can be be reflected using two separate CQL
>> indexes. It is not possible to use modifiers to switch from one to
the other.
> Why not? It seems an eminently sensible way of expressing the
Well if you create an index on a field in Adlib, you explicitly have to
tell what type it is:
 int, date, word, string.
So it's perfectly possible to have a cql.word indexed title, and no
cql.string index available for the same
field. In fact, in the standard app.'s we deliver it's very rare to have
a field indexed on both word and
string. I could reprogram my implementation to use proper CQL semantics,
but then I'd have to throw an
error message on every query that makes me imply cql.string where the
index is actually word, and vice
versa.. And there's no way to report the index type in a ZeeRex record.
I already asked our customer whether
this is a problem for them but didn't hear back from them yet.

>> Adlib interprets terms in the following manner:
>> + operator 'exact': implied modifiers are cql.unmasked [...]
>No, we all agreed that "exact" does _not_ imply unmasked.
Okay, I'll change my implementation. I'll have to add modifier handling
for masked/unmasked selection, then.
I chose these semantics because exact and = do not dictate string resp.
word currently, but it seems
I will have to change that..

>> + operator '=': implied modifiers are cql.masked and either cql.word
>> or cql.string, depending on the index type. This cannot be seen
>> in the explain information but must be described in a profile.
>This is _not_ what "=" means in CQL. It means that the term is
word-structured, irrespective of the index
>being searched, unless overridden by a relation modifier.
See above: that's not how Adlib functions natively, bridging from
correct CQL to Adlib can be done but has
some nasty consequences.

>I disagree, Mike. It means, currently, exact equality, but does not
say how equality is to be determined.
>Unless the term is a list (which may be a singleton) of words.
>Au contraire. This is precisely the difference between "=" and
I guess Mike is right, although I don't like it :-)

>> + operators 'any' and 'all': implied operators are cql.word and
>> The words are combined using OR (for 'any') or AND (for 'all').
>Yup. This was always the intent and should probably be explicit in the
CQL context-set document.
The CQL (1.0) tutorial says so, copying it from there to the CQL context
set might be nice indeed.

>> + adlib.record meta-index: implied operators are cql.string and
>There is nothing in CQL that allows you to infer different
term-structure and masking semantics from an
>index name.
>So you have to treat = "2004-12-25" as a word? or a string?
Sure you can do this.
>Hmm. Your example is strangely provoking.
Another fine point to discuss, just like the URI/word splitting thingy.
Anyway, I'll change adlib.record to
only accept '=', with the correct CQL modifier checks.

>> This implied behaviour will remain intact in future versions, even
>> if modifiers will be supported then.
>Aha! Finally, I spot a tiny, tiny error in the English :-) That
sentence should say "... even if modifiers
>ARE supported then".
Ah, the fine concept of 'tenses'. Quite hard to grasp for the Dutch,
since Dutch is far more liberal in that

>> - operator 'exact' does not imply cql.string, since cql.string or
>> cql.word is index dependent on Adlib.
>The correct way to handle this is to have a single CQL index be mapped
to either one of two different
>underlying Adlib indexes, dependent on whether string or word structure
is used.
Yep, I understand that. See discussion above.
I've extended the profile with a new concept: a prox implementation, as
far as Adlib can implement it. I'm very curious if any of you can accept
me defining it the way I did :-) Adlib can't support any of the prox
functionality as described in the CQL context set..

Then there's still one option question about CQL context set modifiers:
how do I type check number & isoDate searches? I can either infer those
modifiers from the index (which isn't allowed in CQL according to Mike)
or from
the term (which is not foolproof, e.g. string indexes can store numbers
as well). If I want to check all index
types I should check that the modifier is cql.number for an Adlib
integer index, but then users are required to
send =/cql.number for number indexes always. Probably not a nice thing
to do...

Best regards,

Hedzer Westra, Systems Developer

Adlib | Information Systems
Reactorweg 291
3542 AD Utrecht
Postbus 1436
3600 BK Maarssen
tel: +31-30-241 1885

The Adlib Base Profile version 1.0
The Adlib Context Set version 1.0


Initiated: Monday December 13th, 2004
Last updated: Friday December 17th, 2004

Hedzer Westra, [log in to unmask]
Adlib Information Systems


Optional and server-dependent features
Implemented SRW/CQL features
Unimplemented optional features
Adlib server-specific implementations
The Adlib Context set
CQL to Adlib bridging



This document describes the Adlib Base Profile 1.0 of the SRU server that is implemented by Adlib Information
Systems in its Internet Server software. It is only a partial profile; it only describes the software
capabilities. For each installation and configuration of this software a full profile document should be


The URI for the Adlib Base Profile is "info:srw/profile/6/1.0".
The Adlib context set is identified by the URI "info:srw/cql-context-set/6/1.0". The preferred identifier
for this URI is 'adlib'.


The SRW and SRU protocols, and the CQL language, CQL context set and SRW Base Profile are all described on Please refer there for information about this search
& retrieval protocol.


The Adlib Internet Server implements all features required by the SRW Base Profile,
refer to

Optional and server-dependent features

SRW/U and CQL are quite broad protocols which allow for many optional and server-dependent features.
This document defines which optional and server-dependent features are implemented, and how these function.


Adlib supplies a number of applications to its customers, who can make changes and additions to their
applications. Therefore there is no single 'Adlib' context set or profile. Each customer can configure their
Internet Server to produce the SRU context sets and profiles that are required. For any such configuration a
context set and profile document should be created - or existing ones should be configured. This document
describes the Adlib Base Profile and the Adlib Context Set, which form the basis of such a customer-defined
document. Therefore, only meta-indexes are defined here. Metadata formats (like Dublin Core or MarcXML) are
not defined here.

Implemented SRW/CQL features

- protocol version 1.1
- the SRU protocol, i.e., HTTP GET/POST CGI requests
This is a slight extension of SRU, since the current documentation does not mention SRU POST.
- explain operation
- searchRetrieve operation
- CQL 1.1 parsing
- CQL 1.1 handling
- CQL context set as far as the Base Profile requests it, plus some extras:
+ and, or, not booleans
+ prox boolean with an Adlib-specific implementation, see below
+ =, >, <, <=, >=, <> relations
+ exact, all, any, scr relations
+ within relation
+ cql.anywhere, adlib.allIndexes and adlib.record meta-indexes
+ cql.serverChoice surrogate index
- sorting
- surrogate & non-surrogate diagnostics generation
- recordSchemas
- request echoing, xSortKeys and XCQL

Unimplemented optional features

- recordXPath handling
- result sets
- full proximity searches, i.e. the 'prox' boolean
- word anchoring (^)
- matching on a single character using '?'
- scan operation
- SRW (SOAP) as communication layer

Adlib server-specific implementations

- The adlib.allIndexes meta-index searches all indexes defined in the Adlib database at once. This is
different from cql.anywhere, which searches all indexes in all context sets.
- The adlib.record meta-index searches the whole record, so all of the fields in each record. This includes
data that is not indexed (and possibly not even displayed in any record schema) and therefore not
searchable using CQL indexes. The relation must be '='.
Note: In future versions, CQL might support a cql.record meta-index with the same semantics.

- cql.scr is always handled as '='
- The 'within' relation is implemented using range searching. Exactly two words must be supplied, separated
by a single space. The range search type can be selected by using an adlib.range modifier, with the
following values:
inclusive (default)
Example: 'date within/adlib.range=leftexclusive "2000 2004"'
Note: the CQL context set always uses inclusive range searching; there is no range modifier.

- empty term searches are not supported
- * for pattern matching is only usable at the beginning and/or end of a search term

- the prox boolean is accepted only with modifier /<=/0/adlib.record/unordered or
/<=/-1/adlib.record/unordered. These are all defaults, except the unit value: adlib.record, and the
distance of the second example. Other unit and distance values are invalid. 'prox///adlib.record' and
'prox//-1/adlib.record' are the shortest ways to select this option. The left and right arguments of the
prox boolean must be simple searches. See the Adlib section for more explanation.
- Thesaurus-enabled searches can be executed by issuing an adlib.thesaurus modifier with one of the
following values:
These only work correctly on indexes with thesaurus links defined. Otherwise, they fall back on
normal searching. The modifier is supported only for the 'exact' and '=' relations.
- there are two types of CQL context set modifiers: data type modifiers and pattern modifiers.

The accepted data type modifiers are:
Note: cql.uri is invalid

The accepted pattern modifiers are:
cql.unmasked (not yet defined in CQL context set)

Of each modifier type only one can be active for each search clause. The combinations
/cql.masked/cql.number and /cql.masked/cql.isoDate are invalid.

- Adlib interprets terms in the following manner:
Modifier cql.masked is always assumed (unless cql.number or cql.isoDate are supplied).
+ 'exact': implied modifier is cql.string
+ '=', 'any' and 'all': implied modifier is cql.word
For 'any' and 'all' the words are combined using OR (for 'any') or AND (for 'all').
+ 'within': there is no implied data type modifier; there must be two words separated by a single space.

Modifier cql.unmasked means:
The CQL pattern match character * has no special meaning; pattern matching is not possible using
this modifier. The characters ^, ? and * do not have to be escaped.

Modifier cql.masked means:
Pattern matching on * is possible. The characters ^, ?, * and \ must be escaped with \.

Modifier cql.word means:
Words are split and then re-combined using the Adlib separators and concatenators rule.
Word adjacency is not used when searching. An error will be returned when searching with cql.word
on a string index.
Separator characters are: [];,[email protected]()|{}<>? carriagereturn newline space tab
Concatenator characters are: `-=\./~#$%^&_+:"'*
Note: the CQL context set says nothing about how words are to be split but instead leaves that up
to implementations to be specified.

Modifier cql.string means:
Terms are not inspected for separators or concatenators. An error will be returned when searching with
cql.string on a word index.

The difference(s) between the CQL context set is/are the following:
- word adjacency is a feature that Adlib currently does not implement.

- sorting is only supported for hard-wired (case sensitive) paths, not for full XPaths. The customer can
define a path for each CQL index.

The Adlib Context set

The Adlib context set version 1.0 defines:

- adlib.record (whether this will be added to the CQL context set is still unclear, so the prefix is
adlib, not cql)
- adlib.allIndexes

- adlib.thesaurus its six accepted values (generic, broader, etc.)
- adlib.range for the within relation
- adlib.record for the prox boolean
- cql.unmasked (it is assumed this modifier will eventually be added to the CQL context set, hence the
cql prefix)

CQL to Adlib bridging considerations

The special-case prox relation is implemented using the WHEN and WHEN NOT booleans in Adlib. Unit 0 selects
WHEN, unit -1 will select WHEN NOT. In Adlib, the WHEN (NOT) booleans on two distinct indexes will first
search records using the left operand index. For each record the right operand operator and value will be
checked in the same occurrence. The Adlib-specific property of occurrences comes down to the following: each
field can have 0 or more values (unless explicitly stated as non-repeatable), whereas conventional RDBMS's
can only hold 0 or 1 values in a field. Indexes can be specified to index either the first or all
occurrences. The WHEN operator explicitly checks matching occurrences. See section 6.3.10 in the Adlib User
Guide (available from for more information.

This relation is implemented using range searching, i.e. the Adlib WHEN boolean used on two identical

cql.word and cql.string:
Customers creating their own profiles based on this one should clearly state which indexes support word,
string or both. Searches which map to non-existing word or string indexes will return an error, so clients
should be careful to use 'exact' or '=' only when supported. Unfortunately this information cannot be sent
using the explain ZeeRex record and must therefore be documented in a profile.

- The adlib.allIndexes and cql.anywhere meta-indexes might have slow search responses if there are a lot of
- The adlib.record meta-index cannot use any index and will always be slow.
- Searching on *...* is done using the Adlib 'contains' operator, which is slow since no index can be used.

Top of Message | Previous Page | Permalink

Advanced Options


Log In

Log In

Get Password

Get Password

Search Archives

Search Archives

Subscribe or Unsubscribe

Subscribe or Unsubscribe


July 2017
October 2016
July 2016
August 2014
February 2014
December 2013
November 2013
October 2013
February 2013
January 2013
October 2012
August 2012
April 2012
January 2012
October 2011
May 2011
April 2011
November 2010
October 2010
September 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
October 2009
September 2009
August 2009
July 2009
May 2009
April 2009
March 2009
February 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005
May 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004
September 2004
August 2004
July 2004
June 2004
May 2004
April 2004
March 2004
February 2004
January 2004
December 2003
November 2003
October 2003
September 2003
August 2003
July 2003
June 2003
May 2003
April 2003
March 2003
February 2003
January 2003
December 2002
November 2002
October 2002
September 2002
August 2002
July 2002
June 2002
May 2002
April 2002
March 2002
February 2002
January 2002
December 2001
November 2001
October 2001
September 2001
August 2001
July 2001



CataList Email List Search Powered by the LISTSERV Email List Manager