Example on how to setup scheduled indexing

Aug 17, 2011 at 8:46 PM

Hi,

According to http://fulltextsearch.codeplex.com/wikipage?title=Advanced%20Customisation&referringTitle=Home under "Scheduled indexing" it is mentioned that:

 

"In the event that content on your site changes regularly without any nodes being published a web service is provided which allows you to

  • Trigger a full re-creation of the site index
  • Re-index a list of nodes

It is envisaged that this service be called by means of a scheduled task of some kind. No mechanism for doing this is currently provided, though a console application may be added in future."

 

Has anyone tried this and can be able to show an example on how to configure?

Regards,

Martin

Coordinator
Aug 18, 2011 at 9:48 AM

Hi Martin,

The procedure for this should be relatively simple.

1) Download and install the package fulltextwebservices.zip from the download section here

2) If necessary (i.e. if the file isn't already present on your install, it wasn't on mine) download umbraco.webservices.dll, I've also put this in the download section.

3) Modify your umbracosettings.config appropriately, see http://fulltextsearch.codeplex.com/discussions/265853

That should give you the web service, if everything's working correctly you should see it at http://yoursite.com/umbraco/webservices/FullTextService.asmx

In terms of actually calling it, I did start work on a console app to do just that. If you download the latest source code and open the solution you should see a project named "fulltextconsole", which will build a console app you can use, copy the .exe and the .exe.config somewhere after building and modify the .exe.config with your username/password and the correct URL (<endpoint address=".....").

If you schedule that console app to run regularly it should (in theory) trigger reindexing of your site. The console app is poorly tested and probably buggy though, so if you're going to use it in production be sure to test it thoroughly.

Hope that helps.

Bjorn

Aug 18, 2011 at 8:18 PM
Edited Aug 18, 2011 at 8:19 PM

Hi Bjorn,

Thanks for your answer. I can do step 1-3 but since I have no server acces (only FTP) I can´t run an exe file. But a collegue of min pointed me in the direction of making a .NET usercontrol that calls the SOAP webservice. I will try doing that and post my solution if it comes out working good.
Regards,
Martin
Coordinator
Aug 19, 2011 at 9:57 AM

Hi Martin,

The exe doesn't technically need to be run on the server itself, you can run it from any internet connected machine.

If you are writing a .net user control I'd look into calling functions from the FullTextSearch.Admin.AdminActions class directly, rather than faff about with the web service. Just reference the fulltextsearch.dll in your project and call those functions from your user control. That's all the web service actually does. If you look at the code it should be (relatively) simple to figure out what you need to do.

Bjorn

Aug 19, 2011 at 12:44 PM

Hi Bjorn,

 

Cool, thanks. I have now written the .net usercontrol with the following code behind:

 

 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;

namespace FullTextSearchReindex
{
    public partial class reindex : System.Web.UI.UserControl
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            FullTextSearch.Admin.AdminActions.ReindexAllFullTextNodes();
            //FullTextSearch.Admin.AdminActions.RebuildFullTextIndex(); 
            
        }
    }
}

 

 

It works with the ReindexAllFullTextNodes method, but unfortunately the "RebuildFullTextIndex" method gives an error: 

Server Error in '/' Application.

Object reference not set to an instance of an object.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: System.NullReferenceException: Object reference not set to an instance of an object.

Source Error: 

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.


Stack Trace: 

[NullReferenceException: Object reference not set to an instance of an object.]
   UmbracoExamine.UmbracoContentIndexer.GetDataToIndex(XElement node, String type) +164
   Examine.LuceneEngine.Providers.LuceneIndexer.AddNodesToIndex(IEnumerable`1 nodes, String type) +139
   UmbracoExamine.BaseUmbracoIndexer.AddNodesToIndex(String xPath, String type) +93
   UmbracoExamine.BaseUmbracoIndexer.PerformIndexAll(String type) +769
   Examine.LuceneEngine.Providers.LuceneIndexer.IndexAll(String type) +72
   UmbracoExamine.BaseUmbracoIndexer.PerformIndexRebuild() +86
   Examine.LuceneEngine.Providers.LuceneIndexer.RebuildIndex() +348
   FullTextSearch.Admin.AdminActions.RebuildIndex(String index) +83
   FullTextSearch.Admin.AdminActions.RebuildFullTextIndex() +85
   FullTextSearchReindex.reindex.Page_Load(Object sender, EventArgs e) +32
   System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) +14
   System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) +35
   System.Web.UI.Control.OnLoad(EventArgs e) +91
   System.Web.UI.Control.LoadRecursive() +74
   System.Web.UI.Control.LoadRecursive() +146
   System.Web.UI.Control.LoadRecursive() +146
   System.Web.UI.Control.LoadRecursive() +146
   System.Web.UI.Control.LoadRecursive() +146
   System.Web.UI.Control.LoadRecursive() +146
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +2207



Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.1

 

Can you say why this happens?

 

 

Coordinator
Aug 19, 2011 at 1:07 PM

I couldn't say exactly. It looks like examine is throwing the error.

Do you get the same error when clicking the "re-index" button in the full text search tab in the developer section of the back office? Because that esentially calls the same function.

Bjorn

Aug 19, 2011 at 1:49 PM

Hi Bjorn,

Yes, then pressing "Re-Create Index" I get the same error:

 

Server Error in '/' Application.

Object reference not set to an instance of an object.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: System.NullReferenceException: Object reference not set to an instance of an object.

Source Error: 

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.


Stack Trace: 

[NullReferenceException: Object reference not set to an instance of an object.]
   UmbracoExamine.UmbracoContentIndexer.GetDataToIndex(XElement node, String type) +164
   Examine.LuceneEngine.Providers.LuceneIndexer.AddNodesToIndex(IEnumerable`1 nodes, String type) +139
   UmbracoExamine.BaseUmbracoIndexer.AddNodesToIndex(String xPath, String type) +92
   UmbracoExamine.BaseUmbracoIndexer.PerformIndexAll(String type) +769
   Examine.LuceneEngine.Providers.LuceneIndexer.IndexAll(String type) +71
   UmbracoExamine.BaseUmbracoIndexer.PerformIndexRebuild() +86
   Examine.LuceneEngine.Providers.LuceneIndexer.RebuildIndex() +348
   FullTextSearch.Admin.AdminActions.RebuildIndex(String index) +83
   FullTextSearch.Admin.AdminActions.RebuildFullTextIndex() +84
   FullTextSearch.Admin.WebUserControl1.reCreate_Click(Object sender, EventArgs e) +11
   System.Web.UI.WebControls.Button.OnClick(EventArgs e) +118
   System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +112
   System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +10
   System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +13
   System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +36
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +5563

 

Pressing "Re-index" work fine.  I have set <maintenanceServiceUsers>admin</maintenanceServiceUsers> to "admin".

Regaards,

Martin

 

 

Coordinator
Aug 19, 2011 at 2:11 PM

Something is throwing an exception during indexing. I'm not convinced this is necessarily specific to the RebuildFullTextIndex function, it could just be that the exception is being swallowed when indexing is being perfomed through other methods. It's pretty difficult to debug from here though. 

The first thing I would do is to have a look through the umbraco log and see if that sheds any light. That's the umbracoLog table in the site database. Look for records marked error that deal with examine or fulltextsearch and occur when re-indexing is being performed. Post them here. You might also want to check if there are specific pages not appearing in your index (try using LUKE to examine the index directly), as these are likely to be the ones throwing errors.

Bjorn

Aug 19, 2011 at 10:36 PM

Hi Bjorn,

I have triggered index from both backend (the dashboard) and "frontend" (the usercontrol I made for the scheduled job). I throws the following errors:

 

From dashboard:

At /umbraco/dashboard.aspx?app=developer (Referred by: http://www.glrye.dk/umbraco/dashboard.aspx?app=developer): System.NullReferenceException: Object reference not set to an instance of an object. at UmbracoExamine.UmbracoContentIndexer.GetDataToIndex(XElement node, String type) at Examine.LuceneEngine.Providers.LuceneIndexer.AddNodesToIndex(IEnumerable`1 nodes, String type) at UmbracoExamine.BaseUmbracoIndexer.AddNodesToIndex(String xPath, String type) at UmbracoExamine.BaseUmbracoIndexer.PerformIndexAll(String type) at Examine.LuceneEngine.Providers.LuceneIndexer.IndexAll(String type) at UmbracoExamine.BaseUmbracoIndexer.PerformIndexRebuild() at Examine.LuceneEngine.Providers.LuceneIndexer.RebuildIndex() at FullTextSearch.Admin.AdminActions.RebuildIndex(String index) at FullTextSearch.Admin.AdminActions.RebuildFullTextIndex() at FullTextSearch.Admin.WebUserControl1.reCreate_Click(Object sender, EventArgs e) at System.Web.UI.WebControls.Button.OnClick(EventArgs e) at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) at System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) at System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

From frontend/user control
At /4182.aspx (Referred by: http://www.glrye.dk/umbraco/editContent.aspx?id=4182): System.NullReferenceException: Object reference not set to an instance of an object. at UmbracoExamine.UmbracoContentIndexer.GetDataToIndex(XElement node, String type) at Examine.LuceneEngine.Providers.LuceneIndexer.AddNodesToIndex(IEnumerable`1 nodes, String type) at UmbracoExamine.BaseUmbracoIndexer.AddNodesToIndex(String xPath, String type) at UmbracoExamine.BaseUmbracoIndexer.PerformIndexAll(String type) at Examine.LuceneEngine.Providers.LuceneIndexer.IndexAll(String type) at UmbracoExamine.BaseUmbracoIndexer.PerformIndexRebuild() at Examine.LuceneEngine.Providers.LuceneIndexer.RebuildIndex() at FullTextSearch.Admin.AdminActions.RebuildIndex(String index) at FullTextSearch.Admin.AdminActions.RebuildFullTextIndex() at FullTextSearchReindex.reindex.Page_Load(Object sender, EventArgs e) at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) at System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) at System.Web.UI.Control.OnLoad(EventArgs e) at System.Web.UI.Control.LoadRecursive() at System.Web.UI.Control.LoadRecursive() at System.Web.UI.Control.LoadRecursive() at System.Web.UI.Control.LoadRecursive() at System.Web.UI.Control.LoadRecursive() at System.Web.UI.Control.LoadRecursive() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

 

If it might help I would like to use LUKE to analyze the index but I don´t know what LUKE is? Where can I find it?

 

/Martin