ICMP-X Ingest failf half way through

[Sebastian Binz]

Hi,
We are seeing an odd behavior on our ICMP-X and I am hoping someone can point me into the right direction.

Setup:
Barco SP4K with ICMP-X
Synology NAS running DSM 7.1.1 and a DCP Download software. They are connected via a Unifi Network infrastructure. The FTP Server on the synology is one of the available ingest sources of the ICMP.

The new odd behaviour is that ingests from the NAS to the ICMP fail after a rather random amount of time. Smaller Items, such das trailers, will transfer just fine. But feature DCPs will randomly stop somewhere during the transfer at random positions - sometimes after 20GB, sometimes after 50GB or so. Restarting the same feature again will lead to the transfer stopping at another position. Until now I have not found that there is a specific time value that I can attribute to the point where it stops.

Ingesting from another FTP source works like a charm, so this seems to be down to the Synology FTP setup or the LAN setup in between. I have increased the timeout value within the FTP server settings without any change.

In the projectors logfile I can find

 icmp user.err SMS: SMS- Copy failed: (10501) source read error: Transfer stalled: no bytes received for 300 seconds
 icmp user.info SMS: Ingest- Ingest job failed [CPL: TheLifeOfChuck_FTR-1_S_EN-XX_DE_51_4K_TOB_20250217_CPP_IOP_OV (JobId = ef5b941d-67e0-4761-847a-aacf8db664fd)]: CPL copy failed

Has anyone experienced a similar behaviour and found a solution?

[Carsten Kurz]

I assume you use a switch between the ICMP and the Synology. My first try would be to powercycle all involved devices (you did that already), reseat both ethernet cables, and if not successful, get a known/new good ethernet cable and try a direct connection.


I also assume the ICMP-X is running on current software? Which ethernet port of the Barco is used for the ingest (photo?)

[Ryan Gallagher]

I was going to 2nd the known good ethernet cable. As well as seconding the direct test by bypassing any switch and networking directly between synology and the ICMPX ( assuming they both auto-detect cross-over configuration)... and if equipment locations make that doable.

[Ioannis Syrogiannis]

I would focus more to the FTP server's logs than the Alchemy's.
I would also check what flavor of FTP is used. FTP, SFTP, FTPS, etc.
Is the transfer type active or passive?
If active, what ports are open?
Same for passive.
If passive, what is the IP used, internal or external?
What subnet masks are used on the two machines?
On the server, how many are the maximum connections available?

Does the same thing happens when you connect and transfer to your/a computer with -say- FileZilla?
Would the computer be at the same switch/LAN/VLAN with Alchemy?

As there seems to be a time out, I wouldn't suspect the physical connections first. Yet, that would be the simplest explanation.
Then, I would start questioning what might be happening on the switch > router level and going up.
Checking direct connection should rule out a couple of points of failure. Gigabit NICs can handle "cross".

You write that you use a Unifi network. Is that something new? Is the NAS something new? Is that utilization of the connection something new?
Was everything the same a month ago? If yes, was everything working well? If yes, was there any update since?
Could this be relevant: https://wiki.filezilla-project.org/N...on_large_files ?

I understand that I am not of much help here.
It's so many variables that may go wrong, even the „DCP Download software“ could be occupying a port that triggers a connection time out...
Try to limit the extra „players“ and add as you go.

[Marcel Birgelen]

Like Ioannis pointed out, if you have anything like a firewall or device doing NAT between the ICMP-X and the NAS (even the firewall ON the NAS), then there is a chance your FTP TCP control channel gets killed by that firewall/thing in between, because it's been idle for too long. Some clients and servers support "keep alive" messages, being sent down the control channel, however, I haven't found any of those options on either side (Synology DSM / ICMP(-X)).

Have you tried connecting to the FTP on the Synology with another FTP client to see if it exhibits the same behavior?

[Sebastian Binz]

Thank you for all your messages and ideas, it's great to have that much support in such a short period of time.

Re-Boot of all involved devices did not help, nor did new cables. I disabled all firewalls or traffic inspection tools without success.
The Unifi infrastructre was thrown in about 6 months ago and it worked up until recently, when we closed for summer holidays.
However one thing indeed did change, although I would have never guessed it to have any influence: Up until 2 weeks or so the Unifi UDM router used the internet connection provided by an external router (AVM FritzBox). We did throw in a DrayTec Vigor Modem and set up the internet connection to be directly steered by the UDM, which works now with half of the latency.

Apparently what is happening is that the new WAN connecting, using PPPoE is affecting the LAN traffic (although it shouldnt, but that seems to be a common issue on Unifi UDMs, as I was educated by ChatGPT). The standard TCP package Size is a MTU value of 1500. Now the PPPoE connection is using 8 bits of that, leaving 1492 for the actual payload. Normally, this should only happen on WAN traffic, but theres seems to be glitch that also cuts down the size of the LAN traffic and therefore would fragment packages running on 1500 MTU.
I therefore reduced the MTU value within the synology NAS and on the involved switch ports to 1492 and (fingers crossed) that seems to have solved the issue. Additionally, I have moved both devices into the same VLAN to avoid additional processing by the UDM, which could also be a factor.

So in short, the issues seems to have been created by a PPPoE WAN connection which backfires into the package size of LAN traffic. I have not found a way to adjust the MTU size on the Barco size, but it seems to work that the switch is now forcing it.

Which transfer speeds do you typically see in the ICMP-X GUI when ingesting from a FTP source?

[Carsten Kurz]

The Barco ICMP does indeed offer an MTU setting through Communicator. This is mentioned in Barco Commander manual.

Oh my, this PPPoE WLAN MTU adjustment issue is from the stone age. I never thought I would ever see it again.

[Leo Enticknap]

Hmm ... any idea where that is? I can't find it in the obvious places. Series 2:

image.png

I don't have a Series 4 with ICMP available to take a screenshot from, but I can't recall seeing a jumbo packet enable option there, either. I had assumed that they simply allowed anything up to 9K MTUs. In all the ICMP-Xs I've installed in mutiplexes, I've enabled jumbo packets on both the media LAN switch and the content transfer NIC of the TMS computer itself, and there has not been any problem.

However, in this case, if there is a minimum MTU size that the ICMP-X is expecting that packets on this LAN are going below, that's not a situation I've ever had to address.

[Marcel Birgelen]

The minimum MTU of IPv4 is 68 bytes for non-fragmented packets.

First of all, you should, IMHO, never set the MTU on a switch port below 1500, at least if said switch is a Layer 2 switch. There is no way for a layer 2 switch to be a party in MTU sizes negotiated between devices on Layer 3. If the MTU is too low, packets with a larger MTU will simply get discarded. Essentially, you can't set the MTU too high on a Layer 2 switch. (Practically, you can, because many switches suck and will exhaust their buffers at high MTUs...). But, it's not like you suddenly won't fit through the door if you make it wider, if you fitted through before.

As for the term "Jumbo frames" (technically, they're not called packets): any Ethernet frame exceeding 1500 bytes is considered a jumbo frame. Some hardware/software requires you to set a flag to enable "jumbo frame" mode in order to raise the MTU past 1500. In practice, the term "jumbo frame" is a bit more complicated. A VLAN tagged Ethernet frame is usually not called a jumbo frame, although the Layer 2 MTU exceeds 1500 bytes. The same is true for MPLS labeled frames...

I still don't really understand what's happening here. While it's true that many PPPoE connections are still limited to an MTU of 1492, this should only affect your WAN traffic. WAN traffic should get "MSS-clamped" to 1492 bytes, to avoid traffic falling into the PMTU-discovery trap (because even the guys at Google didn't understand how TCP/IP actually works for years...). You should, in this day and age, never have to reduce your local MTU to 1492, if so, something in your network setup has gone pretty darn wrong and you really need somebody to look at it.

Is the NAS on the same subnet as the ICMP-X? Or is it being routed through the Unify UDM?

Also, you really should check if your ISP and router both support RFC4638 (https://datatracker.ietf.org/doc/html/rfc4638)... I know that many ISPs still don't, but heck, that RFC is now almost 20 years old...