 |
 |
Hi all, I received some 'complaints' and suggestions about the Adlib Base Profile from Mike & Rob. Here are my replies. I would like the Base Profile to be final before I start reprogramming, so here are my answers to their remarks. The new Base Profile (Beta-2) has been attached. Many thanks for your help so far! >> - The meta-index cql.anywhere searches all indexes defined in the >> Adlib database at once. It does not search all indexes in all >> context sets, as the CQL context set suggests. [Mike] >It is at best inadvisable, and probably just wrong, to _re_define the meaning of an existing index like >this -- especially such a core one. If cql.anywhere doesn't meet your needs, it would be better to define >your own index that does (or ask to have it added to the CQL set if you think it's of general interest). [Rob] >I think that's exactly what cql.anywhere means. Search all indexes that you know about, but you can't be >expected to search indexes that you don't know about. >Which is (thankfully, for once) exactly how it's defined: >This means "search all indexes from all context sets you know". Okay, okay, you're right. I was just being lazy; iterating *all* indexes is easier than iterating over all CQL-linked indexes. I'll create an adlib.allIndexes index for the current behaviour, and program cql.anywhere like it should be. >> - The adlib.record meta-index searches the whole record. The operator doesn't matter. [Mike] >Perhaps we should consider adding cql.record for whole-record searching (where supported). You can if you wish. I'll implement it for 1.2 if it's added then. >You can't really say "the operator doesn't matter" as this is overriding established semantics of CQL and >the CQL context set. It would be much better to say "the operator must be '=': all others will be rejected". Will do, good suggestion. [Rob] >I'm not sure what exactly happens here. Is it: >(a) You send some terms and it searches the entire record. (And how is this different to cql.anywhere in > practice?) Not all fields are indexed. [Rob] >(b) You send the entire record as a term, and it returns the entire record back to you if it finds it. [Mike] >The former. No-one wants (b). Correct. [Mike] > it is conventional in CQL to talk of "relations" rather then "operators". Unless you have > a compelling reason, you should probably stick to this convention. I'll change that. >> - The Adlib thesaurus operators 'adlib.generic', 'adlib.broader', >> 'adlib.narrower', 'adlib.related', 'adlib.topterm' and >> 'adlib.parents' do thesaurus-enabled searches. These only work >> correctly on indexes with thesaurus links defined. Otherwise, they fall back on '=' searching. [Mike] > Are these relations or relation modifiers? If you don't already have this nailed down, I would recommend > the latter, as they are all refinements on the general relation of equality. Hmm, good suggestion. They're relations now, because 1. they're operators in the Adlib Expert Search language and 2. I was too lazy to implement modifier handling :-) I'll change it to: 'term =/adlib.thesaurus=generic myTerm' etc. if that suits everyone. >There really should be a thesaurus-use context set defined outside of Adlib, for use in this and other >profiles (or the relevant elements should be added to the existing Zthes context set). generic, topterm and parents are rather esoteric operators so I guess bt, nt and rt are just fine. Adlib does use/uf substitution automatically, and has no concept of le, so those three aren't supported as operators. (FYI I'm using 'operator' here because I'm referring to Adlib expert search.) >We actually started this process a month or two back, but got sidetracks -- or maybe mired in excess >complexity. Are there any concrete results from that discussion, or was it just that: a discussion? >Depending on the urgency of your Adlib work, you might try to restart that process and use the resulting >"official" thesaurus-expansion support. I just heard yesterday from our customer that Q1 2005 will be a testing phase, and Q2/Q3 will be the implementation phase so there's enough time to reach an agreement. To top that, the customer probably won't need explicit thesaurus support, I just added it to offer to SRU/CQL as much functionality as the Adlib query language has. >> - The 'encloses' and 'within' operators are implemented using the >> Adlib WHEN operator. Some examples: >> 'term encloses "2000 2004"' translates to 'term >= 2000 WHEN term <= 2004' [Mike] >Nope -- "encloses" is the converse of "within", so > term encloses "2000 2004" >translates to > term <= 2000 WHEN term >= 2004 Erhm that would always be the empty set.. WHEN is just like AND, only a bit different. Read the Adlib manual for details :-) >> 'term within "2001 2005"' translates to 'term > 2001 WHEN term < 2005'. [Mike] >My reading of the CQL context set indicates that this relation is inclusive of endpoints, so you should >translate to term >= 2001 WHEN term <= 2005 [Rob] > dc.date within "2000 2004" > -> Does the record contain a date between (inclusive) 2000 and 2004 I'll add a modifier: within/adlib.range=exclusive (or inclusive - default, rightexclusive, leftexclusive). Makes everybody happy, right? ;-) > foo.rangeOfDates encloses 2002 > -> Does 2002 fall within (inclusive) the date range in the record. > <record> > <dateRange>2000 2004</dateRange> > </record> > would match the encloses query. > (Hopefully this is how your internal queries work) No they don't. I'm a bit confused now about encloses works. In the manual they're not really well defined, if I may say so. If encloses specifically needs a two-dimensional index and a single search term, Adlib can't support it. But Mike's example (the one which results in the empty set) seems to suggest otherwise.. >> - there are two types of modifiers: data type modifiers and pattern >> modifiers [...] [Mike] >This whole section belongs in the CQL context-set document. I know, but since it isn't there I thought I'd just make it explicit in Adlib's profile, so that it's at least defined somewhere.. >> The pattern modifiers are: >> cql.masked >> cql.unmasked (not defined in CQL context set) [Mike] >We should fix that! Something for a 1.2 proposal? >> Note that the CQL context set is not required by the SRW Base Profile! [Mike] >That's not really true, as the CQL context set provides some of the key elements used in pretty much CQL >queries, e.g. the meaning of all the relations. Probably the base profile should make this explicit. Fine by me, I just wanted to make clear that Adlib's implementation violates some of the CQL context set descriptions. Pending this discussion, I'll clear up all or most of these violations. >> The modifiers cql.word and cql.string can not relate directly to >> Adlib term or word matching because this is defined per index by >> the user; in Adlib each index can be either word or term >> indexed. If required, a field can be indexed by term as well as by word. [Mike] >[I don't understand this fully, I think because it assumes you know something about Adlib. Is Adlib's >"term" searching similar to what we mean by "string"?] Yes, it is. I'll try to use CQL terminology in the next version of the profile. >> These two indexes can be be reflected using two separate CQL >> indexes. It is not possible to use modifiers to switch from one to the other. [Mike] > Why not? It seems an eminently sensible way of expressing the difference. Well if you create an index on a field in Adlib, you explicitly have to tell what type it is: int, date, word, string. So it's perfectly possible to have a cql.word indexed title, and no cql.string index available for the same field. In fact, in the standard app.'s we deliver it's very rare to have a field indexed on both word and string. I could reprogram my implementation to use proper CQL semantics, but then I'd have to throw an error message on every query that makes me imply cql.string where the index is actually word, and vice versa.. And there's no way to report the index type in a ZeeRex record. I already asked our customer whether this is a problem for them but didn't hear back from them yet. >> Adlib interprets terms in the following manner: >> + operator 'exact': implied modifiers are cql.unmasked [...] [Mike] >No, we all agreed that "exact" does _not_ imply unmasked. Okay, I'll change my implementation. I'll have to add modifier handling for masked/unmasked selection, then. I chose these semantics because exact and = do not dictate string resp. word currently, but it seems I will have to change that.. >> + operator '=': implied modifiers are cql.masked and either cql.word >> or cql.string, depending on the index type. This cannot be seen >> in the explain information but must be described in a profile. [Mike] >This is _not_ what "=" means in CQL. It means that the term is word-structured, irrespective of the index >being searched, unless overridden by a relation modifier. See above: that's not how Adlib functions natively, bridging from correct CQL to Adlib can be done but has some nasty consequences. [Rob] >I disagree, Mike. It means, currently, exact equality, but does not say how equality is to be determined. >Unless the term is a list (which may be a singleton) of words. [Mike] >Au contraire. This is precisely the difference between "=" and "exact". I guess Mike is right, although I don't like it :-) >> + operators 'any' and 'all': implied operators are cql.word and cql.masked. >> The words are combined using OR (for 'any') or AND (for 'all'). [Mike] >Yup. This was always the intent and should probably be explicit in the CQL context-set document. The CQL (1.0) tutorial says so, copying it from there to the CQL context set might be nice indeed. >> + adlib.record meta-index: implied operators are cql.string and cql.unmasked. [Mike] >There is nothing in CQL that allows you to infer different term-structure and masking semantics from an >index name. [Rob] >So you have to treat dc.date = "2004-12-25" as a word? or a string? Sure you can do this. [Mike] >Hmm. Your example is strangely provoking. Another fine point to discuss, just like the URI/word splitting thingy. Anyway, I'll change adlib.record to only accept '=', with the correct CQL modifier checks. >> This implied behaviour will remain intact in future versions, even >> if modifiers will be supported then. [Mike] >Aha! Finally, I spot a tiny, tiny error in the English :-) That sentence should say "... even if modifiers >ARE supported then". Ah, the fine concept of 'tenses'. Quite hard to grasp for the Dutch, since Dutch is far more liberal in that respect. >> - operator 'exact' does not imply cql.string, since cql.string or >> cql.word is index dependent on Adlib. [Mike] >The correct way to handle this is to have a single CQL index be mapped to either one of two different >underlying Adlib indexes, dependent on whether string or word structure is used. Yep, I understand that. See discussion above. I've extended the profile with a new concept: a prox implementation, as far as Adlib can implement it. I'm very curious if any of you can accept me defining it the way I did :-) Adlib can't support any of the prox functionality as described in the CQL context set.. Then there's still one option question about CQL context set modifiers: how do I type check number & isoDate searches? I can either infer those modifiers from the index (which isn't allowed in CQL according to Mike) or from the term (which is not foolproof, e.g. string indexes can store numbers as well). If I want to check all index types I should check that the modifier is cql.number for an Adlib integer index, but then users are required to send =/cql.number for number indexes always. Probably not a nice thing to do... Best regards, Hedzer Westra, Systems Developer Adlib | Information Systems Reactorweg 291 3542 AD Utrecht Postbus 1436 3600 BK Maarssen tel: +31-30-241 1885 www: http://www.adlibsoft.com