The good news and the bad news on MOSS 2007 search

MOSS 2007 search is a hot topic in our business at the moment and I am confident it will continue to heat up over the year. Many customers and partners have so far been under the impression that Microsoft basically solved all the annoyances we had with SharePoint 2003 search. But some are now starting to hit the wall on some issues. One of these recent issues that has surfaced is wildcard search. MOSS 2007 does to the regret of many still not support this. The search engine does actually support it but the feature is simply not exposed in the new MOSS 2007 search center.

I would like to share a few more unexpected issues that you might not be aware of yet. Well, my intention with this post is actually to try and give you an overview of the improvements Microsoft has to offer in MOSS 2007 search as well as an overview of potential shortcomings that you might encounter.

First of all - I love SharePoint. I also think Microsoft did a really great job improving the SharePoint search engine. It is far superior to the SPS 2003 search engine and can now be compared to the best-of-breed enterpise search engines out there. But the new search user interface still has some serious shortcomings in my eyes.

But let us first have a look at some of the good news you will find in the new MOSS 2007 search engine:

  • Relevancy has been improved significantly.
  • Improved performance and scalability.
  • Continuous propagation of index to search servers during crawl.
  • New Business Data Catalog for indexing LOB systems.
  • Improved meta data management.
  • Duplicate results collapsing.
  • Dynamic summaries.
  • Automatic hit highlighting of search terms in the title, dynamic summary and in the URL.
  • Did you mean?
  • Improved search scopes.
  • Automatic language detection.
  • Option for performing custom security trimming of results.
  • Improved and more consistent API for executing searches.
  • Complete admin API.
  • Improved crawl log.

and some good news to be found in the new search UI:

  • Tabbed search.
  • Boolean search.
  • Improved best bets.
  • People search.
  • Fixed search.

The above list of improvements is not so new anymore - this you can easily find this elsewhere on the net. But the list of shortcomings that I am about to share with you is not something I have seen anywhere else before. So here it is:

  • No wildcard search. As mentioned, the search engine supports it but MS did not expose that feature in the UI.
  • Inconsistent search acoss MOSS and WSS. The problem here is that searching the "This site" search scope in the search box take you to the WSS search page in the _layouts page while the MOSS search scopes take you to the search center. In other words you still have two different search interfaces in MOSS.
  • Lost search history. Searches initiated from the advanced search page are lost on second click.
  • No property value selectors. The advanced search page does not offer a way to let users pick a value from a drop down box on custom properties. It requires the user to know the values a property might assume.
  • Weak support for searching content types. The new content type concept is great but it somehow missed the attention of the search UI team in Redmond.
  • No result actions. Actions like Alert Me and View details that we knew from SPS 2003 search are gone.
  • No result grouping.
  • Poor extensibility of the OOB web parts. The problem here is that all the web parts rely on a hidden object that executes the search. This object is declared internal and is therefore inaccessible for you to extend. One obvious thing many would like to do here is to add support for wildcard search without too much effort. But you will because of this architecture need to replicate the entire search UI in order to just add support for this little feature.
  • No way of easily jumping to the document library where a document lives. It can be annoying to find a document with search and wanting to perform an action on it. To perform the actions you need to go the document library list item representing the document. But there is no link on a search result to do this.
  • No return link. Initiating a search from some site brings you to the MOSS search center. But it does not offer a simple Go Back link allowing the user to return to the site he or she searched from.
  • Searching multiple search scopes no longer works. It worked in beta2 and I would expect this to be a bug that MS is likely to fix in the first SP.

Our product Ontolica for MOSS 2007 will offer a fix for these shortcomings as well as including a lot of new useful features. I will tell you in a later post more about how we have decided to solve the issues above.

Notes: I have now been working with MOSS 2007 search for about a year now. I also worked intensively with SPS 2003 search for the development of Ontolica for SharePoint 2003. I have therefore had a significant interest and need to investigate the improvements Microsoft did on search for WSS V3 and MOSS 2007.

 

Print | posted on Wednesday, January 31, 2007 11:39 PM

Feedback

2/1/2007 11:45 AM
#1

# re: The good news and the bad news on MOSS 2007 search

Great post!

One thing that I can't make is to search for all versions of a document. That is to make the search result include all versions. It only shows the latest (current) one although I have all the access right needed. Most DM systems have a "include all versions" option but not MOSS (?). Is it my poor knowledge of MOSS (probably) or a feature missing in MOSS Search?
2/1/2007 6:20 PM
#2

# re: The good news and the bad news on MOSS 2007 search

Hi Lars Fly Harder,

Thank you for this very comple post about the Ontolica Search features in MOSS. When do you expect to release a first beta?

Regards.
2/1/2007 8:49 PM
#3

# re: The good news and the bad news on MOSS 2007 search

Patrick: Amazing how fast you located my new blog :-) I just got the login the other day and was just about to email you a link to it.

Very good question about searching all document versions. I agree that this is a very good feature for DM's.
I have also heard a request for this feature a few times from some of our customers on Ontolica for SharePoint 2003. I have therefore also looked carefully for this feature in MOSS but also without any luck. I guess I should have included this shortcoming in the list as well.

This feature is unfortunately one of those that is very hard to fix. I guess you probably need to replace the protocol handler in charge of indexing SharePoint sites. I am not sure anyone would want to go there - I certainly don't.

But let us remind Microsoft about this shortcoming and hope they decide to include it in their next release.

2/1/2007 8:57 PM
#4

# Name confusion

Anco: Lars Fly Harder is another Lars in Mondosoft :-) He is one of our great supporters and his real name is "Fly Harder" ;-)

We plan to release a beta of Ontolica Search for MOSS 2007 on February 22nd. But we will already show it to the public Feb. 13th at the SharePoint conference in Berlin. Please come and visit our booth if you plan to go there. It would be great to meet you in person after all our emails last year :-)
2/3/2007 1:52 AM
#5

# re: The good news and the bad news on MOSS 2007 search

Question Lars, Mr.Patrick.T on his blog says you are the guru..can you write up an article on the merits of using MOSS as a Search Applicance, say I had a stand alone WSSv3 server and I want to write a WSS Search web part to query the MOSS server. On teh MOSS Search I define an SSP for a particular crawling scope. Can I offer that as a XMl Query Web Service which the client residing as a WSSv3 Web Part could consume and render the results? Would it be hard to do?
2/3/2007 1:31 PM
#6

# re: The good news and the bad news on MOSS 2007 search

Robert: Yes, you can easily query the SSP using the MOSS search web service. Just add a web reference to http://server/_vti_bin/search.asmx in VS 2005 and use the QueryEx method of the Web Service stub to execute a query. You can also use the GetSearchMetadata method on the stub to obtain the list of available search scopes.
I can recommend you to download the MOSS 2007 SDK from Microsoft - it includes documentation for using the search web service.
PS. We also plan to include support for this scenario in our Ontolica Search product.
2/8/2007 3:23 AM
#7

# re: The good news and the bad news on MOSS 2007 search

Great Post! Yes, you nailed the good and the bad on the MOSS 2007 search. I have discovered the same limitations. I find it fascinating that MS chose to close the hidden object behind the search engine. The weak support for content types and the removal of the "Item Details" link is a msytery to me.
2/19/2007 2:53 AM
#8

# MOSS Search

Well written, objective summary of the strengths and areas for improvement for MOSS search. </steve&gt
3/26/2007 3:41 AM
#9

# re: The good news and the bad news on MOSS 2007 search

Hi Lars

Great post, I was wondering if anyone had come across the problem of adding a new document to a doc library and not being able to find it in search. I was under the impression that a full or incremental search wasn't required if you added a new document it would update the index server service directly and allow users to see the new doc in a search results strait away?
3/26/2007 11:10 PM
#10

# re: The good news and the bad news on MOSS 2007 search

Thanks.
No, an incremental crawl is required to include the document in the index. In MOSS you need to ensure a schedule for the incremental crawl. WSS search will automatically run incremental crawls. A new improvement in MOSS compared to SPS 2003 search is continous propagation of changes to the content index during a crawl. SPS 2003 did not update the index before the entire crawl had completed. Another improvement is the integration with the change log, which SharePoint search hooks into for faster incremental crawls.
4/13/2007 11:16 AM
#11

# re: The good news and the bad news on MOSS 2007 search

Thanks for the info about MOSS 2007 search. Does anyone know the logic behind the "search any of these words" in advance search page of MOSS 2007
4/25/2007 4:00 PM
#12

# re: advanced search


Hi,

After upgrading from Sharepoint 2003 to Sharepoint 2007, the advanced search option is not displaying.

any ideas would be greatly appreciated.

4/27/2007 12:27 AM
#13

# re: The good news and the bad news on MOSS 2007 search

Hi,

Would you know how best bets are being managed in MOSS 2007? I'm doing some research on how best bets are able to improve the clickthrough on our search results page, so I guess I need to know (1) if there's a way that I can track how often best bets are being clicked and (2) if there's a MOSS 2007 way of tracking what results/links would make a good best bet.

Thanks,
Anna
5/2/2007 9:32 AM
#14

# re: The good news and the bad news on MOSS 2007 search

Anna - yes, Best Bets can be managed from the Site Settings menu if you are logged in as a Site Administrator. You will then also find a link to the Usage Statistics reports in the Site Settings menu. It does indeed include a built-in report for tracking best bets. Just remember to enable Usage Analysis in Central Administration and Advanced Usage Analysis on you SSP first.
5/11/2007 12:13 AM
#15

# re: The good news and the bad news on MOSS 2007 search

I cannot understand why Microsoft would make such a change. We implemented an issue tracker type page with many custom columns in 2003. Everyone was so pleased with the searching capabilities that we became very dependant on the site. After the upgrade to 2007 though, the searching capabilities is horable..we can no longer search multi line columns of data. Now I am faced with writing some sort of custom knowledge base to handle our issue tracker database. Thanks a lot MS!!!!
5/29/2007 7:40 PM
#16

# re: The good news and the bad news on MOSS 2007 search

Hi Lars,

Thanks for your reply (item #14). Are you aware of any documention on tracking best bets in MOSS 2007? I've looked at this article -- http://technet2.microsoft.com/Office/f/?en-us/library/5233cf43-6a8f-40cb-9014-0724600e7e381033.mspx. It discusses portal usage reporting, but it didn't mention anything about best bets metrics.

Thanks again,
Anna
6/1/2007 3:01 AM
#17

# re: The good news and the bad news on MOSS 2007 search

How do you utilize the Did you mean technology. I am using the search web service. Is there a way to get it to return the alternate "Did you mean" text so I can present it to the user?

Thanks,
Dan
Dan
6/6/2007 6:22 PM
#18

# Search for first names using search.asmx

Hello Lars,
I'm trying to do a people search using search.asmx. When I use the query below I get an Invalid property exception, could you please let me know what is wrong here.
Thanks
Vinay

Query
SELECT "DAV:href",
"DAV:givenName",
"urn:schemas-microsoft-com:sharepoint:portal:profile:preferredname",
"urn:schemas-microsoft-com:sharepoint:portal:profile:department",
"urn:schemas-microsoft-com:sharepoint:portal:profile:workphone",
"urn:schemas-microsoft-com:sharepoint:portal:profile:workemail",
"urn:schemas-microsoftcom:sharepoint:portal:profile:LastName"
FROM ( TABLE Portal_Content..Scope() UNION ALL TABLE Non_Portal_Content..Scope() )
WHERE
("DAV:contentclass" = 'urn:content-class:SPSPeople') AND
("urn:schemas-microsoft-com:sharepoint:portal:profile:preferredname" LIKE '%o%')
6/14/2007 4:38 PM
#19

# re: The good news and the bad news on MOSS 2007 search

Dan - "Did you mean" suggestions is returned by the SharePoint Search Engine in the "SpellingSuggestion" property on the ResultTableCollection object.
6/14/2007 4:40 PM
#20

# re: The good news and the bad news on MOSS 2007 search

Vinay - Could it be because you forgot a '-' in the property "urn:schemas-microsoftcom:sharepoint:portal:profile:LastName"??

it says microsoftcom instead of microsoft-com
6/15/2007 5:10 AM
#21

# re: The good news and the bad news on MOSS 2007 search

No Wildcard Search? GRRR - on the other hand, Tabbed Search is pretty P.I.M.P
6/21/2007 10:13 PM
#22

# re: The good news and the bad news on MOSS 2007 search

Is there a way to get summary links to show up on searches?
7/17/2007 5:20 AM
#23

# re: The good news and the bad news on MOSS 2007 search

Custom Search:

How to display the summary of returned content with at least 100 characters ....?
8/2/2007 3:07 PM
#24

# re: The good news and the bad news on MOSS 2007 search

I am going crazy with my MOSS2007 trial version I am trying to create a prototype with. I used the default Express SQL 2005.

If I have a Word document with the word "frustrates" in it and search on "frustrate" it won't register a hit. But I get a hit on "frustrates".

When you say there is no wildcard support do you mean it won't do substring searches either? Or do you mean you can't do something like "f?x"?

If anyone has ideas on why the substring search won't work please let me know. I can't see how a search engine wouldn't do substring searches by default...

8/2/2007 5:58 PM
#25

# re: The good news and the bad news on MOSS 2007 search

from your post - No property value selectors. The advanced search page does not offer a way to let users pick a value from a drop down box on custom properties. It requires the user to know the values a property might assume.


Is there a way to get this done in SPS 2007? I am assuming you have this feature available in ontolica?

Thanks
8/3/2007 12:12 AM
#26

# re: The good news and the bad news on MOSS 2007 search

Dano - This has to do with stemming. You must first enable stemming on the Core Results Web Part. You will find a check box for this in the tool part. You must also ensure that you web browser is configured with the preferred language, e.g. en-US. MOSS search reads the list of preferred languages that all web browser provide in the HTTP header on each request. It then picks the first language for stemming.
8/3/2007 12:17 AM
#27

# re: The good news and the bad news on MOSS 2007 search

cloudno9 - No, there is no way to do this OOB in MOSS 2007. You must either code your own advanced search web part (not trivial) or find a thirdparty plug-in. And yes, Ontolica offers this functionality. You can even populate the drop downs from a SharePoint list or a custom web service. We are currently also working on support for AJAX enabled lookup fields, which can be more convienient than a drop-down when you have a large set of property values.
8/3/2007 2:53 PM
#28

# re: The good news and the bad news on MOSS 2007 search

Thanks Lars.
9/13/2007 9:30 PM
#29

# re: The good news and the bad news on MOSS 2007 search

I have extensive experience with SP 2003 but am new to SP 2007. We cannot get the search to crawl attachments in list items. Any idea how to do this in 2007?

In 2003 there was an option to check "Crawl individual list items." This option is absent in 2007.

9/25/2007 7:41 PM
#30

# re: The good news and the bad news on MOSS 2007 search

I'm hoping you can assist me with my current state of befuddlement Lars. I have set up a Sharepoint 2007 site collection and up until last week (or at least I thought), the search scope drop down list always allowed me to search "Entire Site" or the current site/list. Strangely, this week I only have the option of searching "This Site: XXX". My first suspicion is that we had a MOSS license which has expired and we've reverted back to WSS - but I cannot be sure as I'm not the Enterprise site adminstrator. Can someone tell me I'm not going crazy because I can see any reason for this....
Also, I've done a little bit of reading about how to set up a "Search Center" - but I can't find a great deal of information about this on the web. Can anyone point me to where I might get instructions on how to seach up a search center site?


9/27/2007 12:17 PM
#31

# re: The good news and the bad news on MOSS 2007 search

The search box control lists only the current site in the list of sites to be searched. if I have a subsite underneath the home site, then in the home site, search box lists only the home site and not the subsite below it. In fact in 2003 the searchbox control used to display the subsite too. There we had a property ContextSensitiveSearchType enumeration which we could change. I think in 2007, the equivalent is DropDownMode. We tried setting to different values, but there is no effect. Any idea how this could be done?
raj
10/16/2007 12:14 AM
#32

# re: The good news and the bad news on MOSS 2007 search

The advanced search in MOSS appears to find my list items (as well as documents), but the links it returns are bad (i.e. /site/Lists/MyList/123_.000) instead of (/sites/Lists/MyList/DispForm.apsx?ID=123).

Is there any way to have the links in the search result know these are list items, and not documents?
10/23/2007 10:30 AM
#33

# re: The good news and the bad news on MOSS 2007 search

Hi Lars,
Thanks for this shortcomings list.

I was now wondering if Ontolica Search 3.2 solves this shortcoming you pointed out on MOSS 2007:
"Lost search history: Searches initiated from the advanced search page are lost on second click."

My experience in the advanced search of the search center is quite disappointing: when I click the browser's "back" button, I lose the additional "property restrictions" I set for my query. Moreover the values in the listboxes of the 1st remaining property restriction are not the original ones.

So, does Ontolica Search solves this issue ?

Thanks beforehand,
Julien
11/2/2007 7:48 PM
#34

# re: The good news and the bad news on MOSS 2007 search

Also having same issues as Craig "returns are bad (i.e. /site/Lists/MyList/123_.000)". I blew away osearch and spsearch and tried to reinitalize but still get same error.
11/3/2007 10:56 AM
#35

# re: The good news and the bad news on MOSS 2007 search

Hi,
My problem is that i m not able to map boolean type of property from people to meatdata property mappings.
Please help me.
11/5/2007 5:35 PM
#36

# re: The good news and the bad news on MOSS 2007 search

Hi,
I have become your fan.Kewl Article.
I require your help on the following.
I have made a custom search scope.I have Advanced Search webpart in a team site and the core results webpart in a subsite.I went into site settings, top level site collections and made my scope as the default scope.
But when I use the Advanced seaarch webpart i do not get results from my custom scope but from the all sites scope.Is it a bug or am I missing someting..
Please let me know..
11/9/2007 5:17 PM
#37

# Did you mean to return Proper Names

Can anyone tell me how to get "did you mean" to return suggested names? We currently have a site search AND a people search and I would like to know how to link "did you mean" for the people search to our Active Directory or some file that will suggest people's names.

Thanks,
Lis
12/6/2007 12:26 PM
#38

# re: The good news and the bad news on MOSS 2007 search

Hi,
@ramesh i think i had the same problem.
My soultion was to set a default scope in the core result webpart but you have to install a Hotfix otherwise this works only with keyword search (simple search)
KB Article: 937904

greetings
Alexander
12/10/2007 8:58 AM
#39

# re: The good news and the bad news on MOSS 2007 search

Hi,
can anyone refer me the link for searching a free text webpart
12/12/2007 1:55 AM
#40

# re: The good news and the bad news on MOSS 2007 search

I am wondering if it is possible to configure the search so that it will pick up a search term from the middle of a string... Example: Search for liz and it will return elizabeth.

we have tried the wildcard download and it works ok but we have many users that will search for partial words and they will get no results. Its frustrating to search for Rob and get no results because it is listed as Robert... if you could set up the search so that it will find the term withing the String it woudl be better...

I have done this with website stuff on my own but not sure how, if possible, this can be done in SharePoint.


Thanks for any help

Rob
Rob
1/18/2008 10:44 AM
#41

# re: The good news and the bad news on MOSS 2007 search

Question: The search behaves strange, example:

A user came up to me with this issue. There is an excel file in a MOSS document library called M-real_3Q07_update.xls When he tries to search for this file by using the following string: M-real_3Q07_update the search finds nothing, when he uses M-real_3Q07_update.xls the file is found. If I search for real_3Q07_update or _real_3Q07_update the search returns correct results.... Anyone know what the science behind the search is? or is there a best practices search document?
1/21/2008 1:51 PM
#42

# re: The good news and the bad news on MOSS 2007 search

Hi Lars,

A great post on searching capabilities.
And my query is we are planning to integrate SAP Enterprise portal with Share point portal for KMC.But the problem with the integration is SAP-EP uses SAP logon tickets as logon method where as Sharepoint uses kerberos. can you suggest me how can i integrate both.
thanks and regards
sam.
sam
3/6/2008 8:23 PM
#43

# re: The good news and the bad news on MOSS 2007 search

Hi, I want the search to return result from the List items is this possible? please advise.
3/13/2008 2:22 AM
#44

# re: The good news and the bad news on MOSS 2007 search

I have a List in SharePoint that is visible by 3 people... I have tested it out and if we search from the main site those 3 people can find what they search for in the list and nobody else can... GREAT....

The problem is that the list is growing so what they would like to do is search just that list (there is a drop down that allows this)...but when they search just on that list it returns no results...

What am I missing.... we put the same search terms in but on search just that list we dont get a return....but on the whole site we get a return.

thanks in advance

Rob F
3/20/2008 12:27 PM
#45

# re: The good news and the bad news on MOSS 2007 search

Excellent post for the search.Excellent work done on the search.
I have one problem with search.
I am doing search for the document as bellow

Search Result Type = All Result / Document
Where the Property = Emp Last Name = Panchal

it is reruning the two document.

BUT WHEN I TRY BELLOW SENERIO

Search Result Type = All Result / Document
Where the Property = Emp First Name = kapil

i am getting the two documents and the one more document which is the link to the actual folder where this document is loaded.

can any body answer this isuue?

Thanks a lot for the great posts....
4/9/2008 9:32 AM
#46

# re: The good news and the bad news on MOSS 2007 search

Hi,
This Blog is very use full to me to figure out my search problem. Now I am facing problem with

"Individual list items are not displayed in the search results in SharePoint Portal Server 2007"

How should I configure search setting to enable search in list item

Thanks in advance
6/16/2009 10:33 PM
#47

# re: The good news and the bad news on MOSS 2007 search

Lars,

I have 100 page word document, would someone let me know,what is the best way to index anf search the document

Neel
Title  
Name  
Email
Url
Comments   
Please add 4 and 8 and type the answer here: