Interesting article and even more interesting blog. Am reading it from the very beginning, makes me realise I need to start reading up on Big Data – I’m doing the First Derivatives kdb course next month, but perhaps I also need to look at Hadoop

Was in a similar position to this blogger, relying on Resharper to tell me there was an issue, but not really investigating further.

While on the topic of Resharper, it also picksup double check locking (as discussed yesterday) issues

Singleton

Prompted by a question on the double check locking pattern, I decided to do some investigation. I started with looking at Singletons, where the pattern is/was commonly used - I have been Unity for a while now, so was pretty rusty on the best way to do things. Skeet has a comprehensive page, and suggests that double check locking is not the best option. (The answer I gave was to use lock or Monitor with locktaken overload). One alternative in C# 4.0 is to use Lazy<T>.

Skeet’s first example shows how not to do it - (I put a sleep in to show what could go wrong)

and using Lazy (example 6)

code below shows the tasks running in parallel

ConcurrentDictionary

I had originally intended to look at Lazy when looking at the ConcurrentDictionary. People had been burned by assuming methods such as GetOrAdd were thread safe and atomic, which is notcorrect. I had a dictionary with types <string, List<string>>, meaning that I was to insert/update the list if the key already existed. The conventional method, taken from here

highlights the problem

but if we create an extension method

I’m creating a blotter for displaying CDS spreads. The initial functionality is to highlight changing cells, green for an increase, red for decrease. One solution is given here

http://stackoverflow.com/questions/4946912/how-to-animate-the-background-of-a-textblock-when-changing-the-value-of-the-boun

my XAML is shown below

and code

I was wondering if there was an alternative method, as the above seemed a bit painful. My first attempt was to manage everything inside the (row) viewmodel. I had a timer for both bid and ask, and these were fired ~500ms after the bid/ask was updated. The main issue I had here was not freezing the SolidColorBrushes. This solution worked, but was obviously not ideal.

I decided to keep track of changed cells from within the model. When we receive an updated spread, we update the value in the ObservableCollection, and then we check whether the cell was still waiting to be reset. If yes, we update the timestamp to ensure it isn’t reset at the appropriate time, if no, we add it to the list

On startup we create a timer. This will call the RefreshCells method periodically (where the period is a lot shorter than the cell highlight time)

The RefreshCells method is shown below. If the elapsed time is greater than the cellHighlightTime, we call ResetCell on the item within the ObservableCollection and then remove the time from the updatedItems dictionary

Initial attempts at this (always best to do a simple implementation to test future iterations against) shown below

However, looking at our processor usage, we are not fully utilising the capabilities of the machine

so let’s write a simple parallel loop using Parallel.For

looks a lot better from a CPU point of view

but there are a couple of problems….

The first is that “+=” is not thread safe (nor for that matter is ++). So let’s fix that

This fixes that issue, but makes the other issue, namely speed, worse. Using PFX can speed up processing if the logic inside the loop is sufficiently intensive – our simple loop is not. We are way slower than our very first attempt.

The next attempt fixes this. We work out how many cores are available to us, and divide the work equally.

This improves on the original implementation, but we can improve further by removing the “Sum” at the end.

Alternatively, we can use partitions, but this is slower (on my machine)

One alternative (if the iteration count had been lower) would have been to have a look at Enumerable.Range

I raced in this last year with Weybridge but we didn’t do that well so when I was given the opportunity to race again this year (now with Walton), I jumped at the chance. Our crew consisted of novices ranging in age from 15-41, with differing experience levels. We targeted winning the novice categories of the Hyne Cup, the Kingston Head, and if we were lucky, the HORR 2012. The Hyne Cup was pretty straightforward as there were only 2 other boats, ditto the Kingston head –we beat 20 boats in our category – the 2nd place boat was ~40 seconds behind.

The HORR would be a far stiffer challenge – our coach Neil West said before the race that every hundredth of a second counted (something that would become very apparent), and we couldn’t let off until we had crossed the line. We were starting 372nd, which meant waiting in the rain for at least an hour and a half, but time passed relatively quickly as we watched the other competitors go past. Eventually it was our turn – we started well (3:08 - boat 372) and before long passed our first boat. This was a mixed blessing, as while it encouraged us to push hard to catch the next boat, we had to be careful not to use all our energy too soon. I remember desperately wanting to see Hammersmith bridge as I was absolutely exhausted but it took an agonising amount of time to pass under it. We pushed it to the very end, and once past the finish I collapsed in the boat as we drifted under Putney bridge.

The row back is always painful, as you are rowing against a fast flowing tide, but we eventually got to the landing stage, and were informed that we had won (novice) by 2/100 of a second! (over a 19:30 course). This was revised down to 1/100 of a second, and a few days later were were awarded joint victory with Trentham (something we had no complaints about). The awards ceremony was held at London Boat Club, and the pennants handed out by Sir Matthew Pinsent

I had to do it all again the next day – this time with the Novice Vets!

Link here. The most interesting thing for me is Columnstore indexes.

As it says on the website, the package manager is used to automate the process of downloading, installing, upgrading, configuring, and removing packages from a VS Project. Once installed, packages can be installed from the prompt

packages that I’ve downloaded include Unity, log4net, nUnit and RhinoMocks

Once added, the packages are added to a packages.config file

Following on from yesterday’s post on developing a simple RTD COM server, today’s post will show how to do the equivalent in Excel-DNA.

The first task is to download Excel-Dna – I am using version 0.29. Next we copy ExcelDna.xll (or ExcelDna64.xll if developing against 64bit) to a suitable location (I have a release directory), and rename it to something suitable. Next create a text file and give it the same name (and with .dna suffix) as you have done for ExcelDna.dll – see highlighted files below.

We then create a static class (this is where the magic happens) that will be the interface between our sheet and code. We also need to add a reference to ExcelDna.Integration

Below you can see the code – note that we are passing our ICDSQuote identifiers as discussed yesterday – this call will be passed through to ConnectData. “ExcelDNA.RealtimeCreditDataDNA” is the progID of our RTD class

You will recall from yesterday that we had to add Guid, ProgID, and ComVisible attributes to the RTD code – this is no longer required. The IRTDServer is now referencing ExcelDna.Integration.Rtd.

Excel-DNA differs in that we don’t need to Register for COM Interop when debugging,

and don’t need to Register with Regasm (this can be a big bonus if the end machines have limited registry permissions), meaning that we don’t need register/unregister functions (shown below)

Top deploy, all we need to do is copy the xll, the dna file, and the dll, or if we want to combine to one file, we can use ExcelDnaPack.exe – this can be used to merge the dll files into the xll (simple dna file shown below – note we have to specify runtime version if we are using 4.0, and use Pack=true to include a dll (case sensitive)).

The way we use the C# RTD server differs as well.

For the COM server, we add as an Automation server

whereas for Excel-DNA we add via the add-ins

You will recall the SubscribeToCDS method mentioned earlier -  to call from Excel, we call as though it were a conventional function rather than a call to RTD

After implementing, we can see that both our implementations match with BBG

I am writing a simple RTD server that will be used to verify that my (near) real time credit pricer is behaving as it should. The RTD server pushes out CDS spreads onto a spreadsheet and is compared to data pulled direct from BBG (using BDP)

The service uses a WCF duplex channel to subscribe to topics and retrieve updates via the callback. The first step is to implement IRtdServer, generate a GUID, define the ProgId, and set ComVisible to true (not forgetting to also do this in the assembly.info)

We next need to think how we will store our data. Excel has the concept of topics. These are used to identify a unique set of values (per RTD server). So if I subscribed in the sheet to “MSFT” twice, Excel would only pass one topic. We therefore have a one to one mapping from topic to a ICDSQuote (a CDS has ticker, tier, term and quote type (currently bid/ask) to define uniqueness), so we use a BiDictionary. We need to store the available CDS quotes (currently the 5 year constituents in the on the run CDX and ITRAXX (125 constituents in each, and we store bid and ask  - 500 total)). Finally, we have a ConcurrentQueue that holds the updated quotes that have come from the WCF service. We chose a queue as we want the oldest quotes to be overwritten by newer ones (FIFO), and a ConcurrentQueue was used as we will be reading and writing (potentially) at the same time.

The next step is to code the interface methods

• ServerStart

Here we initialise our logging (we don’t have the luxury of being able to use Unity and inject logger etc), set the callback object, and populate available quotes (I would do this in a better way if it were not a prototype). We return 1 if successful. We also subscribe to the event that will send up update CDSQuotes

• Connect

When the user makes a request to a new topic (see above for topic definition), they pass the redcode, tier, term and bid (and also specify the ProgID)

we firstly validate the input (note use of dynamic for return value)

if all is OK, we check that the quote exists in our allquote cache, add to our mapping bidictionary, subscribe to the topic, and retrieve the return value. (taking care to set getNewValues to true (otherwise we get #N/A))

• RefreshData

Here we loop round the concurrentQueue populating a dictionary with the latest spreads. I couldn’t think of a way of populating the 2D array without this interim step unless I resorted to locking (and I think  I then would’ve used the conventional queue and done away with the interim dictionary step).

 

When finished, we must ensure that we set the topic count. Note that we could return all items whether they had been updated or not, but if we are subscribing to many topics, and few are updated per refresh cycle, this is inefficient.

• DisconnectData

When the topic is no longer needed, we can remove from our local mapping biDictionary, and unsubscribe from the WCF service.

 

• Heartbeat

This is called occasionally by Excel when updateNotify hasn’t been called – all we need to do is return 1

• ServerTerminate

We clean up here – dispose of the WCF client class (which unsubscribes to all topics (this is done automatically by Excel by calling DisconnectData, so we are doing this as a backup)

The final piece of code is the event handler. The main thing to note here is that we call UpdateNotify until we are successful – by successful I mean not receiving the “The message filter indicated that the application is busy” exception. This rarely happens with this code as the CDSQuotes only flow through a few times a second, so we don’t call UpdateNotify that frequently.

There is some debate as to whether UpdateNotify should only be called every couple of seconds or alternatively whenever an update comes through. I have chosen the latter for this implementation, but I can easily switch to the former using a timer.

Below is shown the sheet working. The left hand side contains the Bloomberg subscriptions, the right hand side the RTD implementation. Of interest to us are the columns F9:G9 – this shows the difference between the sum of the (500) BBG subscriptions compared to our RTD implementation. (The button just out of view on the right is for RTD debugging – for when the descriptive #N/A is shown!)

References