282 lines
19 KiB
TeX
282 lines
19 KiB
TeX
\documentclass[10pt,a4paper]{article}
|
||
\usepackage[utf8]{inputenc}
|
||
\usepackage[english]{babel}
|
||
\usepackage{amsmath}
|
||
\usepackage{amsfonts}
|
||
\usepackage{amssymb}
|
||
\usepackage{xcolor}
|
||
\usepackage{pifont,mdframed}
|
||
\usepackage{verbatim}
|
||
|
||
|
||
\usepackage{geometry}
|
||
\geometry{
|
||
a4paper,
|
||
total={170mm,257mm},
|
||
left=20mm,
|
||
top=20mm,
|
||
}
|
||
|
||
\usepackage{hyperref}
|
||
\hypersetup{
|
||
colorlinks=true,
|
||
linkcolor=blue,
|
||
filecolor=magenta,
|
||
urlcolor=cyan,
|
||
pdftitle={skyscraper8 Manual},
|
||
pdfpagemode=FullScreen,
|
||
}
|
||
|
||
|
||
\author{Fey}
|
||
\title{skyscraper8 \\ A hobbyist project to learn about MPEG-II}
|
||
\date{October 2025}
|
||
|
||
\NewDocumentCommand{\codeword}{v}{%
|
||
\texttt{\textcolor{blue}{#1}}%
|
||
}
|
||
|
||
\newenvironment{warning}
|
||
{\par\begin{mdframed}[linewidth=2pt,linecolor=red]%
|
||
\begin{list}{}{\leftmargin=1cm
|
||
\labelwidth=\leftmargin}\item[\Large\ding{43}]\textbf{IMPORTANT:}}
|
||
{\end{list}\end{mdframed}\par}
|
||
|
||
|
||
\begin{document}
|
||
|
||
\maketitle
|
||
|
||
Hello! Thanks for your interest in skyscraper8, a program I'm writing to learn about satellite and Cable TV technology, mainly for fun - as a hobby, not out of professionalism.
|
||
|
||
\section{An explanation of what skyscraper8 is}
|
||
Consider a compression format like Pavlov's 7Z, Roshal's RAR, Katz' ZIP, or a container format like GNU TAR. These formats are designed to store one or multiple files in a larger container file and optionally compress these.
|
||
|
||
Now consider a program like 7-zip, WinRAR, or WinZIP. These archiving programs are designed to extract files out of containers mentioned above and many other file container formats.
|
||
|
||
Next, introduce the MPEG-2 Transport Stream\footnote{ISO/IEC 13818-1, ITU-T Recommendation H.222.0}. (hereinafter referred to as "TS") This is a data structure designed to carry video, audio and additional other accompanying data in real-time and for storage. Such additional data may be IP packets, an EPG, an interactive application like a game to play during commercial breaks, receiver system software updates or album art on a radio channel. A broadcaster's imagination is the only real limit here.
|
||
|
||
As you can probably imagine, most of these things I named exist in a filesystem based structure on a computer. Therefore, since the receivers designed to make use of this data have to extract it from a TS, a standard computer should be able to do so as well. However, there are not many applications out there that can actually do this. \\
|
||
A video playback application like VLC or MPC-HC treat a TS as video file, not an archive, and will play it back - rightfully so. \\
|
||
An archiving program like WinRAR also assumes a TS is a video file and isn't even wrong about that. But that means it wouldn't know how to extract contained files from it.
|
||
|
||
This is the niche gap skyscraper8 intents to cover: Extracting files transmitted along in a TS.
|
||
|
||
\section{How to get TS into skyscraper8}
|
||
skyscraper8 is a command-line operated tool. A graphical user interface is planned, but not yet fully developed. \\
|
||
This means that you have to use the command-line interpreter of your operating system to use skyscraper8. On Windows, both cmd.exe and PowerShell are known to work well.
|
||
On Linux and macOS, any halfway recent shell should be fine.
|
||
|
||
\subsection{Using a StreamReader compatible tuner}
|
||
\textbf{This is the preferred way} and applies to tuners like the popular TBS5927 or TBS6903x.
|
||
\\You can use any StreamReader.dll hosting application like CrazyScan or VMA Stream Reader that is capable of streaming to TCP and have skyscraper8 read that stream.
|
||
\\Use your hosting application and start a TCP stream like you normally would. Then call skyscraper8 from a command line:
|
||
\begin{verbatim}
|
||
.\skyscraper8.exe cscan tcp://127.0.0.1:6969
|
||
\end{verbatim}
|
||
|
||
\label{sec:gswarning}
|
||
\subsubsection{Working with GS using StreamReader}
|
||
There are a few caveats when working with StreamReader.dll to capture GS.
|
||
|
||
Some older versions of StreamReader.dll are known to interfere with the STiD135's BBFrame encapsulation feature. Version 1.2.4.101 is a prominent example of doing this. It changes the encapsulation PID and also seems to frame GSE frames differently. Therefore, please make sure to use a recent version of StreamReader.dll. Version 1.2.5.208 is known to work correctly.
|
||
|
||
Also, make sure to disable StreamReader.dll's internal deencapsulator. You can do this by keeping the semicolon in StreamReader.ini like this
|
||
\begin{verbatim}
|
||
;FrameMode=1
|
||
\end{verbatim}
|
||
or set it to 0 like this:
|
||
\begin{verbatim}
|
||
FrameMode=0
|
||
\end{verbatim}
|
||
|
||
Why is that? While StreamReader.dll's Deencapsulation works correctly in theory, it does not seem to check whether an encapsulated packet actually contains valid MPEG-II packets and will happily try to deencapsulate all other packets as well, causing a lot of Packets with invalid data to appear in the outputted TS.
|
||
|
||
|
||
|
||
\subsection{Using a BDA compatible tuner}
|
||
Use TSDuck's \codeword{tsp} and use the \codeword{dvb} input plugin and the \codeword{ip} output plugin.
|
||
Here's how to do it:
|
||
|
||
\begin{enumerate}
|
||
\item Run skyscraper8 from the commandline using the following command:
|
||
\begin{verbatim}
|
||
.\skyscraper8.exe udpin
|
||
\end{verbatim}
|
||
\item Then run \codeword{tsp} like this:
|
||
\begin{verbatim}
|
||
tsp -v -I dvb -a 4 --delivery-system DVB-S2 --frequency 11,523,000,000 \
|
||
--symbol-rate 22,000,000 --polarity horizontal -O ip 127.0.0.1:9003
|
||
\end{verbatim}
|
||
\end{enumerate}
|
||
|
||
\subsection{Using a video4linux compatible tuner}
|
||
See the paragraph about BDA tuners. The same applies to video4linux compatible tuners. TSDuck's \codeword{tsp} can be used on Windows and Linux in the same way.
|
||
|
||
\subsection{Using a SAT\textgreater IP capable tuner}
|
||
This feature is intended for SAT\textgreater IP server devices like the Digital Devices Octopus NET or the Kathrein EXIP 418. If you're using a PCI/PCIe tuner card or an USB Tuner with your PC, these passages about SAT\textgreater IP do not apply to you.
|
||
|
||
To use skyscraper8 with a SAT\textgreater IP device, open up a shell and use the following command scheme:
|
||
|
||
\begin{verbatim}
|
||
.\skyscraper8.exe satip IP_ADDRESS DISEQC POLARITY FREQUENCY SYSTEM SYMBOL_RATE
|
||
\end{verbatim}
|
||
|
||
Replace these variables as follows:
|
||
\\
|
||
\begin{tabular}{llp{5cm}}
|
||
\hline
|
||
IP\_ADDRESS & \parbox{13.5cm}{The IP address (for example 172.20.20.122) of your SAT\textgreater IP server, or "auto" for autodetection of a SAT\textgreater IP server.} \\
|
||
\hline
|
||
DISEQC & \parbox{13.5cm}{The DiSEqC 1.0 Port you want to use. Say "1" here for A/A, "2" for B/A, "3" for A/B, 4 for B/B} \\
|
||
\hline
|
||
POLARITY & \parbox{13.5cm}{Either "H" or "V". The SAT\textgreater IP specification also allows "L" and "R", but this is not yet implemented in skyscraper8.} \\
|
||
\hline
|
||
FREQUENCY & \parbox{13.5cm}{The frequency you want to tune to in MHz. (for example 12226)} \\
|
||
\hline
|
||
SYSTEM & \parbox{13.5cm}{SYSTEM is either "S" or "S2". The SAT\textgreater IP specification does not consider S2X, but in my testings, an S2X capable device like the Octopus NET also accepts "S2" here for S2X transponders. } \\
|
||
\hline
|
||
SYMBOL\_RATE & \parbox{13.5cm}{The symbol rate of the transponder you want to tune to in Ks. (for example 27500)} \\
|
||
\hline
|
||
\end{tabular}\\
|
||
|
||
So a valid command would be for example:
|
||
|
||
\begin{verbatim}
|
||
.\skyscraper8.exe satip 172.20.20.122 2 V 12226 S2 27500
|
||
\end{verbatim}
|
||
to tune to the Hotbird DVB-NIP demo transponder, assuming Hotbird is at DiSEqC B/A. \\
|
||
|
||
If you don't know your SAT\textgreater IP server address, the following command would accomplish the same:
|
||
\begin{verbatim}
|
||
.\skyscraper8.exe satip auto 2 V 12226 S2 27500"
|
||
\end{verbatim}
|
||
|
||
If your SAT\textgreater IP server features a STiD135 chipset, you can also catch GS/GSE with it.
|
||
|
||
\subsection{Using prerecorded TS files}
|
||
This is probably the simplest way to get a TS into skyscraper8. Just run the skyscraper8.exe from the commandline passing the path to your TS file as an argument.
|
||
For example:
|
||
\begin{verbatim}
|
||
.\skyscraper8.exe C:\Videos\capture.ts
|
||
\end{verbatim}
|
||
|
||
If you have a folder containing multiple TS files, you can also pass a directory path as an argument:
|
||
\begin{verbatim}
|
||
.\skyscraper8.exe C:\Videos\Blindscans\
|
||
\end{verbatim}
|
||
This will cause skyscraper8 to process ALL ts files in the specified directory. Note that this will also check subdirectories.
|
||
|
||
Extracted files will be written to the directory the command-line interpreter has been cd'ed to.
|
||
|
||
If you are using Microsoft Windows, you can also Drag’n’Drop a TS file onto the skyscraper8.exe. The extracted data will be placed into the same directory the TS is in.
|
||
|
||
|
||
\label{sec:cannotdo}
|
||
\section{What skyscraper8 can not do}
|
||
\begin{itemize}
|
||
\item Watch TV or Listen to Radio. Extracting files and playing back audio/video are two very different things, and skyscraper8 is not designed to do the latter. If this is something you're looking for, you probably want something like ProgDVB or DVBViewer.
|
||
\item Descramble encrypted channels. This is illegal in most, if not all, jurisdictions, and I want nothing to with that.
|
||
\item Circumvent copy protections. TS contained on Blu-Ray disc are usually scrambled using AACS. Although not too difficult, bypassing it is also a complicated legal matter, which is why I prefer not doing it.
|
||
\item Descramble encrypted DOCSIS\footnote{Data-Over-Cable Service Interface Specifications} Traffic. While skyscraper8 does understand DOCSIS traffic and is perfectly capable of sniffing packets from it using tuners like the TBS-6281 SE, it will drop any scrambled packets - meaning those that have the DOCSIS privacy extension enabled.
|
||
\item Do a professional grade analysis of a TS. There are many intricacies in broadcasting systems which are out of scope for a simple file extractor developed by a mere hobbyist. \hypertarget{bettertools}{If you're looking for a professional grade analyzing tool}, I'd recommend you to check out the tools included in VMA Video Analyzer, Dektec's StreamXpert, GkWare's StreamGuru, or the \codeword{analyze} plugin of TSDuck.
|
||
\item Handle MPEG-DASH streams. Currently, there are two known video formats used in DVB-NIP: HLS and MPEG-DASH. While skyscraper8 is able to extract segments of both types from a TS, I do not know of an easy way to playback MPEG-DASH formatted streams. HLS on the other hand is trivial.
|
||
\item Repair broken GS. When using a non STiD135-based tuner to tune to a GS, the framing gets broken in extremely weird ways. Also, when using StreamReader.dll to tune to a GS with a STiD135 - it is imperative to disable to StreamReader.dll's deencapsulator. It often breaks the framing as well. See \hyperref[sec:gswarning]{the notice about GS in Section 2.1}
|
||
\end{itemize}
|
||
|
||
\section{How skyscraper8 was made}
|
||
skyscraper8 is developed in the C\# programming language, using Microsoft Visual Studio 2022. Therefore it requires a Microsoft .NET runtime, which is included in Microsoft Windows and freely available for most Linux distributions and Apple's macOS.
|
||
|
||
The .NET assembly of skyscraper8 is not obfuscated or protected in any way. This is on purpose. If you want to study how skyscraper8 works under the hood, I hereby allow you to use tools like RedGate's Reflector or JetBrains' dotPeek to inspect the skyscraper8.dll file. I plan to release the clear source code at a later date. \\
|
||
|
||
Although I can't and won't force you to give credit, if you find some of my source code inspiring or helpful and want to use it in your own projects, it would be great if you give credit to where you got it from. \\
|
||
|
||
This document was typeset in \LaTeX{}, using the TeX Live distribution, and while I wrote it myself, I did use GPT-5 for proofreading the initial version of this document. The proofreading of this document is the only part of skyscraper8 for which an LLM was used. No part of the actual skyscraper8 codebase itself was written by an LLM. \\
|
||
|
||
For generating artificial TS recordings to test on, TSDuck was used. \\
|
||
|
||
\subsection{Experiments conducted using TSDuck}
|
||
|
||
\subsubsection {Creating a TS that carried multiple sub-TS using MPE}
|
||
|
||
Since skyscraper8 public release 12, it's possible to have skyscraper8 extract one or more TS carried via Multiprotocol Encapsulation. Unfortunately the dish I have at home is not aimed at satellites carrying such a stream. So I used TSDuck to build one artificially. \\
|
||
|
||
The following script was used to to run \codeword{tsp}:
|
||
|
||
\verbatiminput{../Documentation/TSDuck-Samples/experiment2/run.sh}
|
||
|
||
Contents of tdt.xml:
|
||
|
||
\verbatiminput{../Documentation/TSDuck-Samples/experiment2/tdt.xml}
|
||
|
||
Before executing that script, run this beforehand, to get a stream of Stingray Spa, of course you can use any other IPTV station you'd like:
|
||
|
||
\begin{verbatim}
|
||
tsp -v -I hls https://lotus.stingray.com/manifest/ose-122ads-montreal/samsungtvplus/master.m3u8 \
|
||
-P regulate -P history -O ip 127.0.0.2:3000
|
||
\end{verbatim}
|
||
|
||
For the other stream, I used Offener Kanal Berlin, like so:
|
||
|
||
\begin{verbatim}
|
||
tsp -v -I hls https://alex-stream.rosebud-media.de/bounce/alexlivetv50.smil/index.m3u8 \
|
||
-P regulate -P history -O ip 127.0.0.2:2000
|
||
\end{verbatim}
|
||
|
||
Finally, run the script above to get a clean TS containing two other TS encapsulated in MPE.
|
||
|
||
|
||
\subsection{Personal remarks and some useless bonus information}
|
||
|
||
\subsubsection{Credits}
|
||
skyscraper8 uses some external libraries:
|
||
\begin{itemize}
|
||
\item One of the external libraries is Ionic.Zlib.Core.dll, which is part of the \href{https://github.com/DinoChiesa/DotNetZip-2025}{DotNetZip} tool set by Dino Chiesa.
|
||
\item Also used is Newtonsoft.Json.dll a.k.a. \href{https://github.com/JamesNK/Newtonsoft.Json}{Json.NET} by James Newton-King.
|
||
\item Some of the GS handling code got inspired from the \href{https://github.com/newspaperman/bbframe-tools}{bbframe-tools} written by newspaperman.
|
||
\end{itemize}
|
||
|
||
\subsubsection{How and why skyscraper8 began: A young man's dream}
|
||
Ever since I was a child, I was fascinated by TV. But not actually watching it, rather understanding how it works. I grew up in the 90s, in an area where Cable TV was not really available, and terristial reception (yeah, those TVs with bunny ears!) only worked when it felt like it. So satellite was the best way to get my childish fix of cartoons! We can say the dish on my roof has always been a companion. Far more interesting than actually watching TV was scanning through the printed TV guides. I was curious. How did they make these? And how would I make one myself? I wondered. Of course my parents wouldn't know - they're not techies. \\
|
||
|
||
Of course, as I got older, I eventually moved onto other things. Like programming, trading cards, and video games. But just like with the TV, I eventually got more interested in how these games work, instead of actually playing them. \\
|
||
|
||
I got internet access much later in my life than my peers did, and when I found out that there are forums in which people discuss the technicalities about video games, it blew my mind. I used to be a frequent reader (and occasional contributor) of the XeNTaX\footnote{Unfortunately, that forum is long gone from the internet, but can still be experienced thanks to the amazing work of the people over at the Internet Archive: \url{https://web.archive.org/web/20230925120533/https://forum.xentax.com/ }} forum, from where I learned a lot! \\
|
||
|
||
At the time, my circle of friends was crazy about Yu-Gi-Oh! Online 3, which was an excellent simulation of the trading card game by the same name. I was really bad at this game, not even a mediocre player. I was far more busy studying how the game worked under the hood, and eventually managed to write a program that extracts all the card graphics and the soundtrack of the game. This resulted in my guild leader having the best quality available soundtrack rips on his Youtube account. As you can probably guess, I went on and reverse engineered more and more video games. But as life goes on, you eventually graduate from school, head to university, and then get a job. Interests and people can change over time, and so did I. As I grew older I kind of lost interest in video games. Nowadays I rarely play anymore. \\
|
||
|
||
But what has that to do with skyscraper8? That's easy! One day my parents, who I still stay in touch with asked me to record something off live TV. Up until that point, I totally forgot about the dish on the roof of the house my room is in, but at that point it got relevant again. Upon searching for an easy way to record satellite TV on a PC I stumbbled upon Thierry Lelegard's tsduck. With tsp's plugins, it was real easy to schedule a recording. And since I still had the natural curiosity of a game reverse engineer in me, I opened up the .ts file in a hex-editor, and that's exactly the point in time when the DX virus hit me. After reading a bit of the source code of \href{https://images.videolan.org/developers/libdvbpsi.html}{libdvbpsi}, an idea manifested in my head. And that idea became "skyscraper".
|
||
|
||
\subsubsection{A Brief history of skyscrapers1 to 8}
|
||
At that time had a job of administering some databases. My childhood fantasies of making my own TV guide were now within reach. I had "skyscraper" at home, a bunch of databases at work - and MySQL is free someware. That was shortly before COVID-19 ran wild in the world. During lockdown I had made it: A program that reads a list of frequencies from a database, record a series of TS files, parses all the DVB tables, and writes the results into the database. That was all the first skyscraper could do. It was written in Java, and missed a lot of things.
|
||
|
||
skyscraper5 came about when I changed jobs. I was a software developer at that time. And since we used Java at my day job, and I wanted some distance from my job, I switched my hobbyist projects back to .NET. - .NET 5 to be exact - that's why I called it skyscraper5. That was on the 5th of March, 2022.
|
||
|
||
Eventually, I upgraded to .NET 8, mainly due to dependency issues, and I renamed the project fittingly to skyscraper8. \\
|
||
|
||
If you read up to this point, I wish to thank you from the bottom of my heart. I guess by now you can tell that skyscraper8 is a very personal and dear matter for me. My goal was never to make the best tool for handling TS files, there are \hyperlink{bettertools}{much better options} regarding that, but to satisfy my desire to learn. And let me tell you, I learned a lot during the development of skyscraper8, both as a person and as a programmer.
|
||
|
||
|
||
|
||
\subsubsection{Music!}
|
||
Like a lot of programmers, I do enjoy listening to music while working. Some programmers even put song references in their software. Like how MKVToolnix' version names are actually song names, or how BSD developer fk even put \href{https://www.fabiankeil.de/nutzloseinfos.html}{a list of albums on his website} listing what albums he listened to while making it. Although this is absolutely useless, I'd liketo do this as well. Therefore, here follows a list of musical albums I enjoyed listening to while developing skyscraper8 - no claim to completeness.
|
||
|
||
\begin{itemize}
|
||
\item Bel Canto - White-Out Conditions
|
||
\item Beautiful World - In Existance
|
||
\item Kitaro - Millennia
|
||
\item Macross 82-99 - A Million Miles Away
|
||
\item Megazone 23 Vocal Collection
|
||
\item Moodswings - Moodfood
|
||
\item Opus III - Guru Mother
|
||
\item Satsuki Shibano / Yoshio Ojima - Caresse
|
||
\item SHIFT UP - Stellar Blade Original Soundtrack
|
||
\item Small Affairs - Small Affairs
|
||
\item Starmani Series - la fillette révolutionnaire UTENA
|
||
\item SuganoMusic - EUROBEAT FESTIVAL VOL.7.5
|
||
\item Yoshio Ojima - Hands-Some
|
||
\item Youssou N’Dour - The Guide (Wommat)
|
||
\item Yu-Gi-Oh! Sound Duel Quarter Century Collection
|
||
\end{itemize}
|
||
|
||
\end{document} |