Compellent SAN

We recently took delivery of a Dell/Compellent iSCSI SAN to replace an existing iSCSI SAN. (Dell now own Compellent).

At an architectural level, the Compellent is the first virtualised SAN I’ve used. With a virtualised SAN, you don’t allocated discs to RAID groups. Instead, you give the SAN some guidance on what RAID levels to use and the SAN sorts it all out, moving data blocks between the differing RAID levels based on usage.

Remember I said you don’t allocate discs to RAID groups? The SAN dynamically spreads data across discs using the required RAID level. It’s completely possible for a physical disc to hold RAID 10, 5 & 6 data blocks all at the same time. If you’re used to tightly controlling which discs in which shelf are used, you may have problems letting go!

Another issue of a virtualised SAN is that you can’t just read a simple spec sheet and it tell you the specs of the unit. It all depends on the drives you install in the system. Dell ran a tool against my current SAN to view its usage patterns, then came back with a configuration to at least match its current usage.

Before you go ahead and install your SAN, Dell strongly recommend that you go through their “Pro Deploy” service. This service aims to ensure your SAN is installed & configured correctly for your environment. This includes them sending engineers to site to do the unpacking and racking of the unit. I had two engineers who said they’d been scheduled six hours to do the racking. They did it in about 15 minutes. (I could have done it on my own in under 30 minutes)

It’s in pre-deploy questionnaire they send you that you start to find out the differences with the Compellent.

With my old iSCSI SAN, it had two iSCSI IP addresses and a single management IP address. The Compellent requires a few more addresses. In fact, it requires 10 iSCSI IP addresses and five management IP addresses!

But that’s not all the Compellent wants. It wants internet connectivity back to the Dell mothership. If the Compellent finds a fault it automatically opens a support call with Dell. I can confirm this works as the installing engineer forgot to put the system into maintenance mode when doing a test reboot of the system. Within five minutes I had Dell on the phone asking what was wrong as they’d seen an alert that one of the controllers had gone offline.

The system’s connectivity goes further than a simple email alert, though. If you enable it, Dell can remote into your system. No need for firewall rules. No giving them passwords or stressing how to give them a 2FA token. Just tick the box and the Dell engineer can get straight into your system.

To assign the device’s management IP address, you need to have a Windows machine on the same subnet as the Complement’s management interface. No DHCP here, just good old fashioned broadcasting. (Actually, they say it uses multicast rather than broadcast. Not sure why…) But once you’ve set the device’s IP address you can administer it from outside its subnet.

Then there’s the Dell Storage Manager. It does statistics/log collecting, and also allows acts as a central configuration platform for multiple arrays.

Dell Storage Manager needs a SQL Server instance – and this instance can NOT be stored on the Compellent. The docs say the monitoring system can use either:

  • An in built database engine (suspected to be just a simple flat-file system)
  • MS SQL Express
  • MS SQL Server
  • Oracle MySQL.

The first option is not supported at all other than for testing/demo of the management system.

The SQL Express option is technically possible, but Dell really don’t want you to use it for a live system. (My installation engineer wasn’t happy with me using it and muttered several times about me upgrading to SQL Server proper) Which leaves you needing either a SQL Server or MySQL database.

In addition to the database, you need either a Windows VM or an appliance VM which talks to the database and the Compellent. (Yep, that’s another IP address)

There’s also an add-on to VMWare vCentre for the Compellent. This comes as an appliance (which needs its own IP address!) Unfortunately, the add-on only works with the Flash version of the vCentre interface. (In testing, I also found it unreliable.) This add-on talks to your Dell Storage Manager.

To me, one downside of the Compellent is that you can only configure & manage it from Windows client. Yes, we’re in the 21st contrary but Dell/Compellent still haven’t heard of web browsers yet.

The Complement’s only just gone in. I’ll report back later on what it’s like in anger.

Certificates in CallManager

Starting with CallManger 8, Cisco introduced their “Secure By Design” mantra to CallManager. As its name suggests, the idea is to make CallManager more secure out of the box. An important note is that this does NOT mean CallManager encrypts everything on the wire out of the box. That involves putting CallManager in what’s called mixed-mode and that’s a discussion for another day.

With this new Secure By Design philosophy, CallManager administrators have an extra task to worry about: Security Certificates in CallManager. This task is made more unpleasant by two small details. Firstly, you only worry about it when certificates expire – which is only every couple of years. Secondly, get it wrong, and you potentially lock out all your phones!

With Secure By Design (SBD), the TFTP process cryptographically signs all configuration files served to phones. The phones then check the signature against a list of certificates in their Trust List. If the phones don’t recognise it, they refuse to use the configuration.

So where does this Trust List come from? The TFTP service builds a new one every time it starts. Every time the phone boots, it checks to see if it has the current Trust List. If it doesn’t, it downloads the new one. This trust list is signed by the TFTP’s CallManager certificate. If this trust list signing certificate can’t be validated, then the phone rejects the new trust list.

This is where our conundrum begins: If the TFTP service changes its signing certificate, the phone won’t be able to trust its configuration file signed by the TFTP server, or the trust list produced and signed by the TFTP server.

To help alleviate this problem, Cisco added the Trust Verification Service (TVS). If the phone can’t validate a certificate from its trust list, it then goes and asks the TVS for validation of a certificate. The TVS runs on all cluster nodes running the CallManager process.

The connection to the TVS service is checked from the phone’s existing trust list. The TVS can then validate the new signing certificate and the phone can accept the new trust list.

However, this only works if there is more than one node in the CallManager cluster. e.g. a Publisher/TFTP server and a subscriber running CallManager. If you have just a single node, then you have, what is called, a challenge.

In CallManager 10, Cisco added another certificate to the mix: The ITL Recovery certificate.
This extra certificate is managed separately to the CallManager certificate. The idea is that this ITL Recovery certificate is always in the trust list. If you need to, you can tell CallManager to sign the trust list with this separate ITL Recovery certificate (which the phone already trusts) and the phone can then obtain the new CallManager certificates. Once all the phones have obtained the new trust list, you can re-sign the trust list with the normal CallManager certificate and carry on as normal.

Debugging Cisco IOS ISDN Calls

These techniques work on any Cisco IOS gateway with ISDN-2 or ISDN-30 interfaces, regardless of the protocol used to communicate with CallManager. (MGCP, SIP, etc)

First off, login to the IOS gateway, and enable terminal monitor:

# term mon

To view the layer two connectivity, use debug isdn q921. This will show if you have basic connectivity working to the PSTN exchange.

To view ISDN signalling, use debug isdn q931. Much of the information is fairly readable, but the cause codes are usually just displayed in hex. This Cisco document explains the meaning of the hex codes.

If you have a busy gateway (lots of calls or ISDN lines) there are a couple of techniques you can use to filter the calls.

First off, you can restrict the debugging to just one interface:

# debug isdn q931 int ser 0/0/0:15

Alternatively, you can filter based on calling or called number using the debug cond options.

# debug cond calling 5551234

or

# debug cond called 5551234

You can enter multiple conditions and the gateway will “OR” them, showing the calls that match any of the conditions. Note that the numbers you enter into these debug conditions are the raw numbers seen by the gateway with its communication with the PSTN, before any alterations either by the gateway or CallManager.

I’d suggest configuring your debug conditions first, then enable the isdn q931 debugs – otherwise you’ll not be able to see what you’re typing.

sh debug cond will show you the conditions you’ve configured.

Once you’ve finished debugging, remember to issue a no debug all to stop all debugging.

Another handy command is sh voice calls active which, as its name suggest, shows all the current calls in progress through the gateway. BUT – this will not work on MGCP gateways.

Quagga OSPF On Ubuntu

Enabling Quagga/OSPF on Ubuntu isn’t actually too hard. The devil is in the detail. From my own googling, there are few complete examples of how to do both IPv4 & IPv6.

I assume you already have your Ubuntu box up and running with IPv4 & IPv6

In this example, I’m using OSPF to advertise a /32 address (or a /128 in the IPv6 example) that’s bound to the loopback interface.

IPv4

To bind an address to the loopback interface, you can place the following file in /etc/network/interfaces.d

# /32 Address
auto lo:1
iface lo:1 inet static
 address 192.168.254.1
 netmask 255.255.255.255

You can then use “ifdown lo && ifup lo” to get this to take effect. You can use the command “ip addr show lo” to check that the address has bound.

Next, install the quagga package. That’s easily done via the “apt-get install quagga” command.

Now we have to configure quagga. In the file /etc/quagga/daemons we have to enable zebra (the core module of quagga) and ospfd. My file looks like:

 

zebra=yes
bgpd=no
ospfd=yes
ospf6d=no
ripd=no
ripngd=no
isisd=no
babeld=no

In zebra.conf, we have to specify the IP addresses of all the interfaces.

hostname ubuntu
!
interface lo:1
description /32 Address
ip address 192.168.254.1/32
!
interface eth1
ip address 192.168.100.1/24
!
log file /var/log/quagga/zebra.log

Then in ospfd.conf we configure the OSPF routing protocol.

router ospf
 ospf router-id 1192.168.100.1
 network 192.168.254.1/32 area 0.0.0.0
 network 192.168.100.0/24 area 0.0.0.0
 log-adjacency-changes
 redistribute connected
!
log file /var/log/quagga/ospfd.log

(Re)start quagga (service quagga restart) and…

If you’re using UFW, nothing. The problem is that UFW, by default, will be blocking the multicast traffic that OSPF uses to establish adjacencies. So add a rule to UFW to allow those packets in:

ufw allow from 224.0.0.0/24

If you check your external OSPF routing device, you should now see an adjacency being formed with your Ubuntu/Quagga host and if you check the OSPF routing tables, you should see a route to the IP address on the loopback address. Job done, right? Er, not quite. Try pinging the loopback IP address, and it’ll probably fail. As the final step, you need to enable IP forwarding:

# sysctl net.ipv4.ip_forward=1

will enable the forwarding. Edit /etc/sysctl.conf to make this permanent.

IPv6

IPv6 is very similar. First, add IPv6 to our Ubuntu box’s interfaces. For the ethernet interface, create a file similar to:

iface eth1 inet6 static
 address 2001:DB8::1:1
 netmask 64

And for the loopback interface:

iface lo inet6 static
 address 2001:DB8::2:1 # /128 Address
 netmask 128

Perform the “ifdown && ifup” procedure to activate these addresses (do this from the console for eth1, otherwise you risk sawing off the branch you’re sitting on)

Update the zebra.conf file with our IPv6 addresses:

hostname ubuntu
!
interface lo
description IPv6 /128 Address
ipv6 address 2001:DB8::2:1/128
!
interface lo:1
description /32 Address
ip address 192.168.254.1/32
!
interface eth1
ip address 192.168.100.1/24
ipv6 address 2001:DB8::1:1/64
!
log file /var/log/quagga/zebra.log

Enable ospf6d in /etc/quagga/daemones and create a configuration file (ospf6d.conf) for it:

interface eth1
ipv6 ospf6 instance-id 0

interface lo
ipv6 ospf6 instance-id 0

router ospf6
 !Despite this being an IPv6 routing daemon, it still
 !uses an IPv4 address for its ID.
 router-id 192.168.100.1
 interface eth1 area 0.0.0.0
 interface lo area 0.0.0.0
 redistribute connected

log file /var/log/quagga/ospf6d.log

Restart quagga, and…. Yes nothing again. We have to allow IPv6 multicast through. So let’s just do the obvious command:

# ufw allow in from ff02::/16

And…That doesn’t work. Checking our ufw logs, we see that packets are still being blocked. What gives? It seems that UFW is only allowing TCP & UDP, but in the IPv6 world, OSPF runs over a different protocol, strangely called “ospf”. UFW allows us to specify the protocol, so we can just do:

# ufw allow in proto ospf from ff02::/16

and we get the message:

# ERROR: Unsupported protocol 'ospf'

I tried various tricks, but I couldn’t get UFW to allow the IPv6 OSPF packets through using its syntax. Instead, I had to edit the /etc/ufw/before6.rules file and add the lines:

# allow Link local multicast
-A ufw6-before-input -p ospf -d ff02::/16 -j ACCEPT

This *MUST* be placed before the final “COMMIT” line in the file!

A simple “ufw disable && ufw enable” gets this change loaded and it’s all good.

Note that IPv6 forwarding is enabled by default on Linux.

CallManager Upgrade Options

There are three ways to perform a CallManager:

  • Prime Collaboration Deployment
  • GUI Installer
  • CLI Installer

Prime Collaboration Deployment

PCD is Cisco’s latest recommended way for upgrading CallManager. Instead of having to manage a SFTP server and login to every node in your cluster, PCD can store and serve the ISO to all the cluster nodes. PCD can control the upgrade not just of CUCM but CUPS too. PCD can also handle the cluster wide switch version.  Finally, PCD can handle COPs too.

So what’s not to like? To start with, progress information is lacking. It can be hard to tell what’s really go on. PCD also isn’t particularly helpful if there’s a problem with the installation. To find out what’s wrong, you have to fire up RTMT and download the install logs files. Also, PCD can’t handle complex switch version configurations. i.e. Switch these subscribers, then these, then these.

GUI Installer

Until PCD came along, the GUI installer in the CMPlatform web interface was probably the installer people used by default. It’s simple and it does the job. The GUI installer has the nice feature that you can watch the installation. This allows you to see the progress of the installer and quickly see what happens if things go wrong.

Its big downside is performance. Its real-time view of the installer can consume CPU on the web browser. And if you have the installer running in multiple tabs, you can kiss goodbye to doing anything else on that computer.

CLI Installer

The CLI installer is the least friendly. (Although the CUCM installer isn’t exactly complicated) It’s accessed by SSHing onto your CUCM server and issuing the command “utils system upgrade”. Its major advantage is performance: From one computer you can SSH into multiple CUCM servers and run upgrades without suffering huge CPU slowdowns. You can also use GNU Screen to start off the upgrades and then logout.

There’s another minor reason for getting to know the CLI installer: Sometimes Cisco releases patches or COP files that can only be installed via the CLI installer. (This is especially true of UCCX)

 

CallManager MTPs

CallManager has a choice of MTPs available. It’s not always clear the difference between the types or why you might need one.

Reasons to use an MTP

MTPs can be required for any of the following reasons:

  • To provide Early Offer support for SIP call. (Although since CUCM 10.0 this may not be true)
  • To provide DTMF transcoding. (But this may not actually be true of all MTPs)
  • To act as an RSVP agent for calls.
  • To act as a trusted relay point for calls.
  • To provide IPv4/IPv6 conversion for RTP streams.
  • Repacketisation of a stream. (e.g, convert between 10ms & 20ms packetisation)
  • Act as a fixed point in the media stream when one endpoint in the call doesn’t support mid-call media changes. (e.g. Transfers)

MTPs do not convert between codecs. Well, not always…

When not to use an MTP

  • When the call might involve video. Standard MTPs do not pass video.
  • When you need to convert between codecs. In this case you need a transcoder. However, a transcoder can also perform all the functions of an MTP.

MTP Types

There are three different types of MTPs in CallManager and they have differing capabilities.

CUCM MTP

This is the simplest MTP available. It is built into the CallManager software. Each node in a CallManager cluster can only run one instance of an MTP. Each MTP instance can only support a maximum of 48 concurrent calls. It also cannot make any changes to the media (e.g. DTMF conversion). Cisco’s latest recommendation is to only have this as a last resort and use one of the other types instead.

IOS Software MTP

The IOS software MTP runs on a Cisco IOS (or IOS-XE) router with the voice feature set. It is similar to the CUCM MTP but can handle up to 500 concurrent calls. (The maximum number of calls is dependent on the router hardware)

IOS Hardware MTP

This type of MTP requires DSP resources being allocated on the router. So the maximum number of calls is dependant on the available DSP resources.

This MTP can:

  • Change DTMF formats. e.g. “read” the audio stream and extract DTMF signals and generate out-of-band signalling.
  • Re-packetise the audio. e.g. Convert between 10ms & 20ms packetisation

Whilst MTPs cannot convert between codecs, Cisco’s docs claim a hardware MTP can convert between G.711 a-Law and G.711 u-Law.

Configuring IOS MTPs

Both software & hardware IOS MTPs register to CUCM via SCCP.

IOS Software MTPs

!Change to the appropriate value
voice-card XX/XX/XX
 !Assign the DSPs to being in a dspfarm.
 dsp services dspfarm
!
!Adjust based on which CUCM servers this device is supposed to be registering to
sccp ccm 192.168.0.1 identifier 1 priority 3 version 7.0 
sccp ccm 192.168.0.2 identifier 2 priority 2 version 7.0 
!
!Activate SCCP
sccp
!
dspfarm profile 1 mtp
  maximum sessions software 500
  associate application SCCP
  no shutdown
!
sccp ccm group 1
  associate ccm 1 priority 1
  associate ccm 2 priority 2
!Change the name to something appropriate. e.g. MTP_HW_GW-1.
!Note that CUCM has a limit as to the length of this name
!The profile number is that specified in the dspfarm profile command.
  associate profile 1 register XXXX

IOS Hardware MTPs

Hardware MTPs are configured almost the same as a software MTP. However, instead of:

dspfarm profile 1 mtp
  maximum sessions software 500

You put:

dspfarm profile 1 mtp
  maximum sessions hardware X

Where X is the number of DSP resources you want assigned to the hardware MTP.

Hardware and Software MTPs

Having hardware MTPs configured does not preclude having software MTPs configured on the same IOS device. Just make sure they are configured as separate dspfarm profiles and both are listed in the SCCP CCM GROUP section:

dspfarm profile 1 mtp
  maximum sessions software 500
  ...
!
dspfarm profile 2 mtp
  maximum sessions hardware 4
  ...
!
sccp ccm group 1
...
  associate profile 1 register IOS-SW-MTP
  associate profile 2 register IOS-HW-MTP

G.711 a-Law

Note that if you’re in Europe and use G.711 a-Law, you need to add the following two lines to the dspfarm profile command as it defaults to G.711 u-Law:

dspfarm profile 1 mtp
!
 !IOS Defaults to G7.11 u-Law
 no codec g711ulaw
 codec g711alaw

Using MTPs

Regardless of type, MTPs are assigned to devices (phones, SIP Trunks, Gateways, etc) via Media Resource Group Lists.

Unfortunately, CUCM is not clever enough to know what type of MTP is required for each call scenario. As IOS Hardware MTPs are generally quite limited resources compared to the two software MTPs, the sysadmin as to be prudent in how they are assigned. This will mean configuring differing MRGLs containing the different MTP types.

If there are insufficient (or incorrect types of) MTPs to support a call, the call may progress but without the (correct) MTP. This will result in the call progressing but with errors in the media. In the best case, the callers may not notice anything. In the worst, the call will establish but have no audio, or audio will drop during hold, transfers, or have one-way audio, or have DTMF events fail, etc.