Hi all,

I received some 'complaints' and suggestions about the Adlib Base
Profile from Mike & Rob. Here are my replies. I would like the Base
Profile to be final before I start reprogramming, so here are my answers
to their remarks. The new Base Profile (Beta-2) has been attached.

Many thanks for your help so far!

>> - The meta-index cql.anywhere searches all indexes defined in the
>>   Adlib database at once. It does not search all indexes in all
>>   context sets, as the CQL context set suggests.
[Mike]
>It is at best inadvisable, and probably just wrong, to _re_define the
meaning of an existing index like 
>this -- especially such a core one. If cql.anywhere doesn't meet your
needs, it would be better to define 
>your own index that does (or ask to have it added to the CQL set if you
think it's of general interest).
[Rob]
>I think that's exactly what cql.anywhere means.  Search all indexes
that you know about, but you can't be 
>expected to search indexes that you don't know about.
>Which is (thankfully, for once) exactly how it's defined:
>This means "search all indexes from all context sets you know".
Okay, okay, you're right. I was just being lazy; iterating *all* indexes
is easier than iterating over all
CQL-linked indexes. I'll create an adlib.allIndexes index for the
current behaviour, and program cql.anywhere
like it should be.

>> - The adlib.record meta-index searches the whole record. The operator
doesn't matter.
[Mike]
>Perhaps we should consider adding cql.record for whole-record searching
(where supported).
You can if you wish. I'll implement it for 1.2 if it's added then.

>You can't really say "the operator doesn't matter" as this is
overriding established semantics of CQL and 
>the CQL context set.  It would be much better to say "the operator must
be '=': all others will be rejected".
Will do, good suggestion.

[Rob]
>I'm not sure what exactly happens here. Is it:
>(a)  You send some terms and it searches the entire record.  (And how
is this different to cql.anywhere in 
>     practice?)
Not all fields are indexed.

[Rob]
>(b)  You send the entire record as a term, and it returns the entire
record back to you if it finds it. 
[Mike]
>The former.  No-one wants (b).
Correct.

[Mike]
> it is conventional in CQL to talk of "relations" rather then
"operators".  Unless you have 
> a compelling reason, you should probably stick to this convention.
I'll change that.

>> - The Adlib thesaurus operators 'adlib.generic', 'adlib.broader',
>>   'adlib.narrower', 'adlib.related', 'adlib.topterm' and
>>   'adlib.parents' do thesaurus-enabled searches. These only work
>>   correctly on indexes with thesaurus links defined. Otherwise, they
fall back on '=' searching.
[Mike]
> Are these relations or relation modifiers?  If you don't already have
this nailed down, I would recommend 
> the latter, as they are all refinements on the general relation of
equality.
Hmm, good suggestion. They're relations now, because 1. they're
operators in the Adlib Expert Search 
language and 2. I was too lazy to implement modifier handling :-)

I'll change it to: 'term =/adlib.thesaurus=generic myTerm' etc. if that
suits everyone.

>There really should be a thesaurus-use context set defined outside of
Adlib, for use in this and other 
>profiles (or the relevant elements should be added to the existing
Zthes context set).  
generic, topterm and parents are rather esoteric operators so I guess
bt, nt and rt are just fine. 
Adlib does use/uf substitution automatically, and has no concept of le,
so those three aren't supported as 
operators. (FYI I'm using 'operator' here because I'm referring to Adlib
expert search.)

>We actually started this process a month or two back, but got
sidetracks -- or maybe mired in excess 
>complexity.
Are there any concrete results from that discussion, or was it just
that: a discussion?

>Depending on the urgency of your Adlib work, you might try to restart
that process and use the resulting 
>"official" thesaurus-expansion support. 
I just heard yesterday from our customer that Q1 2005 will be a testing
phase, and Q2/Q3 will be the 
implementation phase so there's enough time to reach an agreement. To
top that, the customer probably won't
need explicit thesaurus support, I just added it to offer to SRU/CQL as
much functionality as the Adlib 
query language has.

>> - The 'encloses' and 'within' operators are implemented using the
>>   Adlib WHEN operator. Some examples:
>>   'term encloses "2000 2004"' translates to 'term >= 2000 WHEN term
<= 2004'
[Mike]
>Nope -- "encloses" is the converse of "within", so
>        term encloses "2000 2004"
>translates to
>        term <= 2000 WHEN term >= 2004
Erhm that would always be the empty set.. WHEN is just like AND, only a
bit different. Read the Adlib manual for details :-)

>>   'term within "2001 2005"' translates to 'term > 2001 WHEN term <
2005'.
[Mike]
>My reading of the CQL context set indicates that this relation is
inclusive of endpoints, so you should 
>translate to term >= 2001 WHEN term <= 2005
[Rob]
> dc.date within "2000 2004"
> -> Does the record contain a date between (inclusive) 2000 and 2004
I'll add a modifier: within/adlib.range=exclusive (or inclusive -
default, rightexclusive, leftexclusive).
Makes everybody happy, right? ;-)

> foo.rangeOfDates encloses 2002
> -> Does 2002 fall within (inclusive) the date range in the record.
> <record>
>  <dateRange>2000 2004</dateRange>
> </record>
> would match the encloses query.
> (Hopefully this is how your internal queries work)
No they don't. I'm a bit confused now about encloses works. In the
manual they're not really well defined, 
if I may say so. If encloses specifically needs a two-dimensional index
and a single search term, Adlib 
can't support it. But Mike's example (the one which results in the empty
set) seems to suggest otherwise..

>> - there are two types of modifiers: data type modifiers and pattern
>>   modifiers [...]
[Mike]
>This whole section belongs in the CQL context-set document.
I know, but since it isn't there I thought I'd just make it explicit in
Adlib's profile, so that it's at 
least defined somewhere..

>>   The pattern modifiers are:
>>    cql.masked
>>    cql.unmasked (not defined in CQL context set)
[Mike]
>We should fix that!
Something for a 1.2 proposal?

>>   Note that the CQL context set is not required by the SRW Base
Profile!
[Mike]
>That's not really true, as the CQL context set provides some of the key
elements used in pretty much CQL 
>queries, e.g. the meaning of all the relations.  Probably the base
profile should make this explicit.
Fine by me, I just wanted to make clear that Adlib's implementation
violates some of the CQL context set 
descriptions. Pending this discussion, I'll clear up all or most of
these violations.

>>   The modifiers cql.word and cql.string can not relate directly to
>>   Adlib term or word matching because this is defined per index by
>>   the user; in Adlib each index can be either word or term
>>   indexed. If required, a field can be indexed by term as well as by
word.
[Mike]
>[I don't understand this fully, I think because it assumes you know
something about Adlib.  Is Adlib's 
>"term" searching similar to what we mean by "string"?]
Yes, it is. I'll try to use CQL terminology in the next version of the
profile.

>>   These two indexes can be be reflected using two separate CQL
>>   indexes. It is not possible to use modifiers to switch from one to
the other.
[Mike]
> Why not?  It seems an eminently sensible way of expressing the
difference.
Well if you create an index on a field in Adlib, you explicitly have to
tell what type it is:
 int, date, word, string. 
So it's perfectly possible to have a cql.word indexed title, and no
cql.string index available for the same
field. In fact, in the standard app.'s we deliver it's very rare to have
a field indexed on both word and 
string. I could reprogram my implementation to use proper CQL semantics,
but then I'd have to throw an 
error message on every query that makes me imply cql.string where the
index is actually word, and vice 
versa.. And there's no way to report the index type in a ZeeRex record.
I already asked our customer whether 
this is a problem for them but didn't hear back from them yet.

>>   Adlib interprets terms in the following manner:
>>   + operator 'exact': implied modifiers are cql.unmasked [...]
[Mike]
>No, we all agreed that "exact" does _not_ imply unmasked.
Okay, I'll change my implementation. I'll have to add modifier handling
for masked/unmasked selection, then.
I chose these semantics because exact and = do not dictate string resp.
word currently, but it seems
I will have to change that..

>> + operator '=': implied modifiers are cql.masked and either cql.word
>>     or cql.string, depending on the index type. This cannot be seen
>>     in the explain information but must be described in a profile.
[Mike]
>This is _not_ what "=" means in CQL.  It means that the term is
word-structured, irrespective of the index 
>being searched, unless overridden by a relation modifier.
See above: that's not how Adlib functions natively, bridging from
correct CQL to Adlib can be done but has 
some nasty consequences.

[Rob]
>I disagree, Mike.  It means, currently, exact equality, but does not
say how equality is to be determined. 
>Unless the term is a list (which may be a singleton) of words.
[Mike]
>Au contraire.  This is precisely the difference between "=" and
"exact".
I guess Mike is right, although I don't like it :-)

>>   + operators 'any' and 'all': implied operators are cql.word and
cql.masked.
>>     The words are combined using OR (for 'any') or AND (for 'all').
[Mike]
>Yup.  This was always the intent and should probably be explicit in the
CQL context-set document.
The CQL (1.0) tutorial says so, copying it from there to the CQL context
set might be nice indeed.

>>   + adlib.record meta-index: implied operators are cql.string and
cql.unmasked.
[Mike]
>There is nothing in CQL that allows you to infer different
term-structure and masking semantics from an 
>index name.
[Rob]
>So you have to treat dc.date = "2004-12-25" as a word? or a string?
Sure you can do this.
[Mike]
>Hmm.  Your example is strangely provoking.
Another fine point to discuss, just like the URI/word splitting thingy.
Anyway, I'll change adlib.record to 
only accept '=', with the correct CQL modifier checks.

>>   This implied behaviour will remain intact in future versions, even
>>   if modifiers will be supported then.
[Mike]
>Aha!  Finally, I spot a tiny, tiny error in the English :-)  That
sentence should say "... even if modifiers 
>ARE supported then".
Ah, the fine concept of 'tenses'. Quite hard to grasp for the Dutch,
since Dutch is far more liberal in that 
respect.

>>   - operator 'exact' does not imply cql.string, since cql.string or
>>     cql.word is index dependent on Adlib.
[Mike]
>The correct way to handle this is to have a single CQL index be mapped
to either one of two different 
>underlying Adlib indexes, dependent on whether string or word structure
is used.
Yep, I understand that. See discussion above.
 
I've extended the profile with a new concept: a prox implementation, as
far as Adlib can implement it. I'm very curious if any of you can accept
me defining it the way I did :-) Adlib can't support any of the prox
functionality as described in the CQL context set..

Then there's still one option question about CQL context set modifiers:
how do I type check number & isoDate searches? I can either infer those
modifiers from the index (which isn't allowed in CQL according to Mike)
or from
the term (which is not foolproof, e.g. string indexes can store numbers
as well). If I want to check all index
types I should check that the modifier is cql.number for an Adlib
integer index, but then users are required to
send =/cql.number for number indexes always. Probably not a nice thing
to do... 

Best regards,

Hedzer Westra, Systems Developer

Adlib | Information Systems
Reactorweg 291
3542 AD Utrecht
Postbus 1436
3600 BK Maarssen
tel: +31-30-241 1885
www: http://www.adlibsoft.com



The Adlib Base Profile version 1.0
 and
The Adlib Context Set version 1.0

Beta-2

Initiated: Monday December 13th, 2004
Last updated: Friday December 17th, 2004

Hedzer Westra, [log in to unmask]
Adlib Information Systems
http://www.adlibsoft.com


Contents

 Introduction
 URIs
 SRW, SRU and CQL
 Requirements
 Optional and server-dependent features
 Configuration
 Implemented SRW/CQL features
 Unimplemented optional features
 Adlib server-specific implementations
  Indexes
  Relations
  Terms
  Modifiers
  Sorting
 The Adlib Context set
 CQL to Adlib bridging
 
-------------


Introduction

This document describes the Adlib Base Profile 1.0 of the SRU server that is implemented by Adlib Information 
Systems in its Internet Server software. It is only a partial profile; it only describes the software 
capabilities. For each installation and configuration of this software a full profile document should be 
defined.


URIs

The URI for the Adlib Base Profile is "info:srw/profile/6/1.0". 
The Adlib context set is identified by the URI "info:srw/cql-context-set/6/1.0". The preferred identifier 
for this URI is 'adlib'.


SRW, SRU and CQL

The SRW and SRU protocols, and the CQL language, CQL context set and SRW Base Profile are all described on 
http://www.loc.gov/z3950/agency/zing/index.html. Please refer there for information about this search 
& retrieval protocol.


Requirements

The Adlib Internet Server implements all features required by the SRW Base Profile, 
refer to http://www.loc.gov/z3950/agency/zing/srw/base-profile.html. 


Optional and server-dependent features

SRW/U and CQL are quite broad protocols which allow for many optional and server-dependent features. 
This document defines which optional and server-dependent features are implemented, and how these function.


Configuration

Adlib supplies a number of applications to its customers, who can make changes and additions to their
applications. Therefore there is no single 'Adlib' context set or profile. Each customer can configure their
Internet Server to produce the SRU context sets and profiles that are required. For any such configuration a
context set and profile document should be created - or existing ones should be configured. This document 
describes the Adlib Base Profile and the Adlib Context Set, which form the basis of such a customer-defined
document. Therefore, only meta-indexes are defined here. Metadata formats (like Dublin Core or MarcXML) are 
not defined here.


Implemented SRW/CQL features

- protocol version 1.1
- the SRU protocol, i.e., HTTP GET/POST CGI requests
  This is a slight extension of SRU, since the current documentation does not mention SRU POST.
- explain operation
- searchRetrieve operation
- CQL 1.1 parsing 
- CQL 1.1 handling
- CQL context set as far as the Base Profile requests it, plus some extras:
  + and, or, not booleans
  + prox boolean with an Adlib-specific implementation, see below
  + =, >, <, <=, >=, <> relations
  + exact, all, any, scr relations
  + within relation
  + cql.anywhere, adlib.allIndexes and adlib.record meta-indexes
  + cql.serverChoice surrogate index
- sorting
- surrogate & non-surrogate diagnostics generation
- recordSchemas
- request echoing, xSortKeys and XCQL 
   

Unimplemented optional features

- recordXPath handling
- result sets
- full proximity searches, i.e. the 'prox' boolean
- word anchoring (^)
- matching on a single character using '?'
- scan operation
- SRW (SOAP) as communication layer

   
Adlib server-specific implementations

Indexes
- The adlib.allIndexes meta-index searches all indexes defined in the Adlib database at once. This is
  different from cql.anywhere, which searches all indexes in all context sets. 
- The adlib.record meta-index searches the whole record, so all of the fields in each record. This includes
  data that is not indexed (and possibly not even displayed in any record schema) and therefore not 
  searchable using CQL indexes. The relation must be '='. 
  Note: In future versions, CQL might support a cql.record meta-index with the same semantics.
  
Relations
- cql.scr is always handled as '='
- The 'within' relation is implemented using range searching. Exactly two words must be supplied, separated 
  by a single space. The range search type can be selected by using an adlib.range modifier, with the 
  following values:
   leftexclusive
   rightexclusive
   exclusive
   inclusive (default)
  Example: 'date within/adlib.range=leftexclusive "2000 2004"'
  Note: the CQL context set always uses inclusive range searching; there is no range modifier.
  
Terms
- empty term searches are not supported
- * for pattern matching is only usable at the beginning and/or end of a search term

Modifiers
- the prox boolean is accepted only with modifier /<=/0/adlib.record/unordered or 
  /<=/-1/adlib.record/unordered. These are all defaults, except the unit value: adlib.record, and the
  distance of the second example. Other unit and distance values are invalid. 'prox///adlib.record' and
  'prox//-1/adlib.record' are the shortest ways to select this option. The left and right arguments of the 
  prox boolean must be simple searches. See the Adlib section for more explanation.
- Thesaurus-enabled searches can be executed by issuing an adlib.thesaurus modifier with one of the 
  following values:
   generic
   broader
   narrower
   related
   topterm
   parents
  These only work correctly on indexes with thesaurus links defined. Otherwise, they fall back on
  normal searching. The modifier is supported only for the 'exact' and '=' relations.
- there are two types of CQL context set modifiers: data type modifiers and pattern modifiers.

  The accepted data type modifiers are:
   cql.string
   cql.word
   cql.isoDate
   cql.number
   Note: cql.uri is invalid
   
  The accepted pattern modifiers are:
   cql.masked
   cql.unmasked (not yet defined in CQL context set)
  
  Of each modifier type only one can be active for each search clause. The combinations 
  /cql.masked/cql.number and /cql.masked/cql.isoDate are invalid.
   
- Adlib interprets terms in the following manner:
  Modifier cql.masked is always assumed (unless cql.number or cql.isoDate are supplied).
  + 'exact': implied modifier is cql.string
  + '=', 'any' and 'all': implied modifier is cql.word
    For 'any' and 'all' the words are combined using OR (for 'any') or AND (for 'all'). 
  + 'within': there is no implied data type modifier; there must be two words separated by a single space.
  
  Modifier cql.unmasked means:
   The CQL pattern match character * has no special meaning; pattern matching is not possible using 
   this modifier. The characters ^, ? and * do not have to be escaped.

  Modifier cql.masked means:
   Pattern matching on * is possible. The characters ^, ?, * and \ must be escaped with \.
   
  Modifier cql.word means:
   Words are split and then re-combined using the Adlib separators and concatenators rule.
   Word adjacency is not used when searching. An error will be returned when searching with cql.word
   on a string index.
   Separator characters are: [];,[email protected]()|{}<>? carriagereturn newline space tab
   Concatenator characters are: `-=\./~#$%^&_+:"'*
   Note: the CQL context set says nothing about how words are to be split but instead leaves that up
   to implementations to be specified.
   
  Modifier cql.string means:
   Terms are not inspected for separators or concatenators. An error will be returned when searching with 
   cql.string on a word index.
   
  The difference(s) between the CQL context set is/are the following:
  - word adjacency is a feature that Adlib currently does not implement.
  
Sorting
- sorting is only supported for hard-wired (case sensitive) paths, not for full XPaths. The customer can 
define a path for each CQL index.


The Adlib Context set

The Adlib context set version 1.0 defines:

(meta-)indexes:
- adlib.record (whether this will be added to the CQL context set is still unclear, so the prefix is
   adlib, not cql)
- adlib.allIndexes 

modifiers:
- adlib.thesaurus its six accepted values (generic, broader, etc.)
- adlib.range for the within relation
- adlib.record for the prox boolean
- cql.unmasked (it is assumed this modifier will eventually be added to the CQL context set, hence the
   cql prefix)


CQL to Adlib bridging considerations

prox:
The special-case prox relation is implemented using the WHEN and WHEN NOT booleans in Adlib. Unit 0 selects
WHEN, unit -1 will select WHEN NOT. In Adlib, the WHEN (NOT) booleans on two distinct indexes will first 
search records using the left operand index. For each record the right operand operator and value will be 
checked in the same occurrence. The Adlib-specific property of occurrences comes down to the following: each 
field can have 0 or more values (unless explicitly stated as non-repeatable), whereas conventional RDBMS's 
can only hold 0 or 1 values in a field. Indexes can be specified to index either the first or all 
occurrences. The WHEN operator explicitly checks matching occurrences. See section 6.3.10 in the Adlib User 
Guide (available from http://www.adlibsoft.com/) for more information. 

within:
This relation is implemented using range searching, i.e. the Adlib WHEN boolean used on two identical 
indexes. 

cql.word and cql.string:
Customers creating their own profiles based on this one should clearly state which indexes support word, 
string or both. Searches which map to non-existing word or string indexes will return an error, so clients
should be careful to use 'exact' or '=' only when supported. Unfortunately this information cannot be sent
using the explain ZeeRex record and must therefore be documented in a profile.

Performance:
- The adlib.allIndexes and cql.anywhere meta-indexes might have slow search responses if there are a lot of 
indexes.
- The adlib.record meta-index cannot use any index and will always be slow.
- Searching on *...* is done using the Adlib 'contains' operator, which is slow since no index can be used.