Archive

Posts Tagged ‘tutorial’

How to Install ThingSpeak in Ubuntu 16.04

December 7th, 2016 4 comments

Last week-end I installed ESPurna open source firmware with MQTT server on Sonoff POW WiFi switch, and the next step is find a way to draw power consumption charts in some web based interface. We could do this in the IoT cloud with services like Xively or ThingSpeak, but since one of the goals of replacing the default firmware was not to rely on a proprietary cloud based solution, I decided to find a way to draw those chart in a local server, and it so happens that ThingSpeak is also open source with the code available on Github. Hardware platforms like NanoPi NEO / NEO Air, or Orange Pi Zero boards appear to be particularly well suited for the task of running an MQTT broker and Thingspeak, but at first I wanted to install ThingSpeak in my own Ubuntu 16.04 computer to have a try.

Click to Enlarge

Click to Enlarge

As you can see from the screenshot above I manage to do it, but it requires a bit more efforts than expected, as the project has not been updated since 2015, and does not work out of the box with the latest operating system.

I used various resources on the web including the instructions on Github, as well as this ThingSpeak script for Ubuntu 14.04, and a few other resources.

First we have to make sure Ubuntu 16.04 is fully upgraded:

Ubuntu 16.04 comes with Ruby 2.3, but we need the older Ruby 2.1.0 version for ThingSpeak, so let’s uninstall ruby to avoid conflicts:

Now we can install dependencies, Ruby 2.1.0, and Bundle:

Once this is done, we can get ThingSpeak source code and install it:

This looked successful so I moved on to database configuration:

It’s recommended to change the username and password in config/database.yml for test, development and production databases with your own for security purpose. Once it’s done, let’s try to create the databases:

Sadly it starts with an error:

So I checked mysql2 version and upgraded it to see if it would fix the issue:

The previous error is gone, but only to be replaced by a new one…

Finally, I found out (can’t find where anymore) that I had to edit Gemfile in ThingSpeak directory, and add an older version to mysql2:

Let’s update mysql2, and try to create the databases again:

Damn a permission error. I could not find a proper fix, so at this point the title of the post should possibly become “How NOT to install Thingspeak in Ubuntu 16.04”, as although it will work, the steps below makes the installation insecure since I simply give full databases’ access to thing user. But that will do since I’ll only use it in my LAN, and maybe somebody will point to a secure solution to the issue.

[Update: Thanks to Arthur, I’ve got a more secure solution . I’ve left both insecure and secure workaround for reference, but obviously you should use the secure one, especially it’s not hard]

Insecure (don’t use it, I just left it to show what you should not do):

Secure way (strongly recommended):

This time I can create the databases for Thingspeak:

So now we can go to the next step to load the database with some data required by Thingspeak to work:

Great! Yet another error:

After spending a while for a solution I eventually found it in Rails Github with the reason being that MySQL 5.7 used in Ubuntu 16.04 does ot allows for NULL key.

We’ll need to create config/initializers/abstract_mysql2_adapter.rb file with:

Then we need add the following line at the end of config/environment.db

and run the command again:

Success! Finally…

The final step is to start the server:

Now start your web browser and you can access your local Thingspeak installation @ http://localhost:3000.
I’ll now have to study a little more about Thingspeak, install MQTT, as well as one of the MQTT to Thingspeak bridges available on the web, and see if I can plot power consumption data there.

Getting Started with Pine64 PADI IoT Stamp – Part 2: Serial Console, GCC SDK, Flashing & Debugging Code

November 28th, 2016 5 comments

PADI IoT Stamp module powered by Realtek RTL8710AF ARM Cortex M3 WiFi SoC is a potential competitor to Espressif ESP8266 modules.  Pine64, the manufacturer of the module, sent me their kit with a $2 IoT stamp, a breakout board, a USB to TTL debug board and a J-Link debug board. In the first part of the review I’ve shown the hardware and how to assemble PADI IoT stamp kit. In the second part I’m going to write a tutorial / getting start guide showing how to control the board with AT commands, build the firmware with GCC SDK, and finally demonstrate how to flash and debug the firmware with the J-Link debugger.

The Quick Start Guide indicates you need to connect the USB to TTL debug board to UART2 instead of UART1 as I did on the very similar B&T RTL-00 RTL8710AF module, and set connection settings to 38400 8N1. This did not work for me, and I had indeed to connect the USB to TTL board to UART0 instead (GB0 & GB1 pins).

Click to Enlarge

Click to Enlarge

I’ll be using a Ubuntu 16.04 (Linux) computer for this quick start guide, but you can work with Windows and Mac OS X too, as tools as available for all three operating systems. So in my case I configure minicom to 38400 8N1 using /dev/ttyUSB0 device, and the boot log is almost the same as B&T RTL-00 with the same ROM version and toolchain:

There are however some changes, and for example the firmware used on PADI IoT Stamp has slightly more heap available. The guide also mentions ATS? should show all command available, but it’s not working for me:

Typing “help” as I did with RTL-00 module does not work either, and that does not look since documentation appears to be wrong again, but that’s not a big deal since we have all AT commands listed in that document. I could configure it as “IoTSTAMP” access point:

and enable the HTTP server with ATSW AT command:

It rebooted the IoT stamp with the same WiFi setting, and I could connect to its demo web page for configuration.

Click to Enlarge

Click to Enlarge

Since everything is so similar to B&T RTL-00 I’ll just point out to the post “Getting Started with B&T RTL-00 RTL8710 Module – Serial Console, AT Commands, and ESP8266 Pin-to-Pin Compatibility” for more tests with different AT commands. I still tried to turn on and off the a GPIO pin using the ATSG command since it’s something I did not do with RTL-00:

The first line pull GC0 pin to high level (3.3V), while the second command brings it down to low level (0V). Details about ATSG command:

I did not connect an LED, but instead measured the value with multimeter and could confirm the voltage level was right in both cases.

B&T provided an SDK which required a an unlicensed / pirated version of IAR ARM Workbench, but Pine64/Realtek have released a GCC SDK that do you require you to use pirated software. You can download sdk-ameba-rtl8710af-v3.5a_without_NDA_GCC_V1.0.0 (198 MB) directly from Pine64 website. After unzipped the SDK you can enter sdk-ameba-rtl8710af-v3.5a_without_NDA_GCC_V1.0.0 directory, and open readme.txt to have a look at RTL8710 GCC SDK structure:

Since I only aim to write a getting started guide I won’t go through all of it, but we can see the low level source code & binary, some documentation, an example project, and some tools include Android and iOS apps, OTA download server and more.

Nevertheless the readme.txt tells us to first read “UM0096 Realtek Ameba-1 build environment setup – gcc.pdf” in order to setup our development environment. The instructions are available with Windows and Linux, but again I’m only test them using Ubuntu 16.04. They’ll be very similar since you’ll rely on cygwin in Windows, and if you run the latest Windows 10 you should be able to install Windows subsystem for Linux, and use the Linux instructions.

First you have to make sure some tools and libraries are installed:

then we can build the sample project:

If everything goes well the log should end showing “Image manipulating” as follows:

We can find the application in application/Debug/bin directory:

There’s also an ota.bin image which might be usable using OTA firmware update documentation, but for this guide I want to use the J-Link debugger that the company sent me instead. The GCC SDK is not for PADI IoT stamp, but instead for Realtek Ameba Arduino board, and you’ll be asked to connect the board through one of the micro USB port. That won’t work with IoT stamp since there’s no USB port at all, and instead you’ll need to go and back forth between multiple documentation, and connect the board as per the JTAG/SWD connections diagram shown below.

padi-iot-stamp-jlink-swd-connectionThat document also mentions that:

Required external power VCC 3.3V, JTAG/SWD didn’t supply power to the PADI IoT Stamp, VCC connection from PADI IoT Stamp is used by JTAG/SWD as voltage reference only.

At first, I did not see that, and used it without external power supply, but since I was not successful with the J-Link debugger (for another reason), so I ended up inserting PADI IoT stamp into a breadboard and added Ywrobot power board to provide an external 3.3V power source.

Click to Enlarge

Click to Enlarge

I also soldered a 22uF capacitor, since I’ve read it’s not optional, as it may affect WiFi connection due to power issue. Once I complete the wiring, I connected the debugger to my computer:

There are two sets of instructions in UM96 document to download and flash the code: OpenOCD/CMSIS-DAP and JLink, so since I had a J-Link debugger, I went with that latter. First you have to download J-Link Software and Documentation pack and for my system I selected ” Linux, DEB Installer, 64-bit V6.12″. After accepting the EULA, I got JLink_Linux_V612_x86_64.deb file which I installed as follows:

Now we can start JLink GBD server for a Cortex-M3 as explained in the document:

So the JLink debugger is detected, but failed to connect to the target. Apart from the last error, everything looks exactly as in the documentation. That’s when I started to add an external power boar, solder the capacitor, and double check my connection. But finally after many trials and errors, I realized that I had to use a SWD connection (SWCLK/SWDIO signals) instead of JTAG…

Now keep the GDB server running, open a new terminal windows in the same directory (where you’ve built the code), and run make flash to download and flash the code to the board:

There will be a lot of message as above, and the GDB Server windows will show its own set of messages:

Now if you want to debug your code, you can run make debug to start the gdb console:

At this point, you’ll just need to use gdb command out of the scope of this post, but you can find tutorials online, for example this. You can also run make ramdebug in order to write ram_all.bin to RAM then enter gdb debug.

So that’s only the debug part, but if you want to create your own application, you’ll need to study the source code, and there are plenty of examples to help you in project/realtek_ameba1_va0_example/example_sources folder:

Note that this is only useful is you want to use PADI IoT stamp as a standalone module, and if you connect it to another board (e.g. Arduino) you can control it through the AT command set.

So while PADI IoT stamp is a usable platform with its GCC SDK, currently documentation is not always correct, and development should be reserved to experienced developers, as it’s not exactly as straightforward as Arduino, Lua or other firmware often used in ESP8266. Arduino will most likely never supported on IoT stamp due to memory constraints, but mbed support should come to the module in the first part of next year, which will make everything much easier.

If you want to go further, you can read the documentation on PADI IoT stamp resource page and the GCC SDK, checkout rebane’s openocd example, and/or read a forum post about controlling IoT stamp through Pine A64 board using Python.

If you want to play with your own, you can get PADI IoT stamp for $1.99, the breakout board kit for $0.5, the USB to serial debug board for $1.99, and the JLink (SWD) debugger is $7.99 on Pine64 online store. Please note that the two debug boards are standard components, so you may use your own, if you already have such hardware.

How to Create a Bootable Recovery SD Card for Amlogic TV Boxes

November 19th, 2016 7 comments

I reviewed Rikomagic MK22 TV box about two weeks ago, and with the firmware I had, online firmware update was not enabled, and the company only released .IMG firmware for Amlogic USB Burning Tool, a windows only tools that’s not well designed, and requires some procedure that vary slightly from boxes to boxes which in some cases forces to buy a male to male USB cable.

I’ve now started reviewing R-Box Pro TV box also based on Amlogic S912 processor, online firmware update is not working either, and again I only managed to find .IMG firmware for the box on GeekBuying. However, I’ve been informed that “USB Burning Tool” firmware can now be flashed through a micro SD card, or USB flash drive with all recent Amlogic TV boxes, so I’ve tried this method instead of R-Box Pro, and decided to report my experience in this post following some instructions on Freaktab made by user Calc. I’ll show instructions in Linux (which could be further streamlined), and then Windows.

Linux Method

After downloading and extracting the rar’ed firmware file (twice), I ended up with a single img firmware files (aml_s912_q6330-R-BOX-PRO-3gddr-mac-20161015.img).

First we’ll need to have a tool to extract some files from the firmware. Create aml-upgrade-package-extract.c with the code below which I found on Freaktab too and slightly modified it to parse the firmware filename:

Now compile the tool with gcc:

and run the thing on the firmware file:

It will extract a bunch of files:

You’ll just need aml_sdc_burn.ini and aml_sdc_burn.UBOOT, plus the IMG file itself to create a bootable mass storage device.

Now find the device for your micro SD card (formatted with FAT32) with lsblk:

In my case it’s /dev/sdd, but I’ll use /dev/sdX in the instructions below, just change the command with yours.

and finally re-mount the SD card/flash drive, copy the firmware file and aml_sdc_burn.ini to the root of the device, and rename the firmware to aml_upgrade_package.img to match the string in aml_sdc_burn.ini:

Now make sure no other USB devices or SD card are inserted in the TV box, and insert your bootable (micro) SD card or USB flash drive into the TV box. If your TV box is fully bricked, you have nothing to do, and the update should start straightaway, but if it is partially bricked or just working fine, you still need to press the recovery button, apply power, and release the button in order to enter recovery mode. The upgrade should then start automatically as shown below.

micro-sd-card-firmware-update

Patiently wait for the update to complete and you should be all good. Please note that I first tried with a USB flash drive, and the method did not work. Once the update is complete, you’ll see the “Android success” logo.

amlogic-firmware-update-successAt this point, remove the micro SD card, and power cycle the board to complete the final steps of the update, and within one or two minutes you should get the Android launcher. If instead the firmware ends with “Android failure” logo showing a red cross, verify your firmware MD5 (e.g. with md5sum in Linux) to make sure it’s not corrupted. If it is, re-download the file and/or re-copy the file to the micro SD card.

Windows Method

If you are a Windows user it’s much easier as you just need to use Burn_Card_Marker tool, which you can download here. The documentation show the interface as shown below.

burn_card-maker-english

But after starting the program in Windows 7, and changing language with the top menu, second option to English, it stayed in Chinese language and looked like the screenshot below instead.

burn_card_maker_v2-0-2

[Update: After changing the Language to English, select the third option in the top menu to apply the change…amlogic-card-maker-menu

…restart the app, and it will be in English.]

It’s not really a big issue, but you can still select your SD card device (F: drive in my case), load the file, and press “Make” button to start create a bootable recovery (micro) SD card.
amlogic-create-bootable-card

You’ll get a “Success!” pop-up window and the end, and the content of the SD card will show the firmware, and the two aml_sdc_burn files.

burn-card-maker-files

Now you can insert the micro SD card in the TV box, enter recovery menu with the pin hole or other method for your TV box, and firmware update will start automatically.

Good Luck!

Getting Started with RabbitMax Flex IoT and Automation Hat for Raspberry Pi

October 30th, 2016 No comments

At the beginning of the month I shows how to assemble RabbitMax Flex, a Raspberry Pi HAT compliant add-on board for Raspberry Pi boards with 40-pin header, that targets IoT and home automation project with its relay, IR transmitter and receiver, I2C headers for sensors, buzzer, RGB LED, and more.  Since I’ve already described the hardware, I’ve spend some time this week-end following the user’s guide to play around with the board using a Raspberry Pi 2 board, and try various features.

rabbitmax-flex-demo

The user’s manual explains that you need the latest version of Raspbian, but I’d not played with my Raspberry Pi 2 board for a while, so the kernel and firmware were quite old:

So the first thing I had to do was to upgrade Raspbian. There are basically two options to upgrade, either downloading and dumping the latest Raspbian firmware image to your micro SD card, and update it from the command line, for example through SSH, and I went with the latter what :

This took several hours on my board, so in hindsight it may not have been the best options. In order to complete the update, I had to reboot the board, and could confirm the Linux kernel and Broadcom firmware had both been updated:

Now we can install I2C tools and vim in order to play with RabbitMax Flex sensors:

We also need to enable I2C though raspi-config:

and go to Advanced->A7 I2C to enable I2C.

raspberry-pi-i2c-configuration

We need to reboot the board again to make sure I2C drivers are automatically loaded at boot time.

RabbitMax Flex user’s manual recommends to use a USB to serial debug at this stage, but it’s just as simple, and IMHO more convenient, to execute command using an SSH terminal. So let’s carry on with software installation with some more dependencies and wiringPi library to control GPIOs:

The system configuration is now complete, and we can use some code samples provided by Leon Anavi, RabbitMax Flex’s developer:

There are eight samples in the directory available in C (using wiringPi library) and/or Python (using python-rpi.gpio):

  • button – Sample to detect when the button is released
  • lcd – Displays “RabbitMax rabbitmax.com” on the LCD display (C language only)
  • rgb-led – Changes RGB LED color every two seconds in loop between red, green and blue
  • sensor-light – Displays BH1750 light sensor intensity in Lux (C language only)
  • buzzer – Turns on the on-board buzzer (beep sample), and plays Star Wars music (starwars sample).
  • relay – Turns on the relay for 3 seconds, and turn it off
  • sensor-humidity – Prints temperature & humidity values from HTU21D I2C sensor (C language only)
  • sensor-temperature – Prints temperature & pressure values from BMP180 I2C sensor (C language only)
RabbitMax Flex LCD Sample

RabbitMax Flex LCD Sample

The first sample I tried was the C code for the button:

WiringPi requires to run the code as root/sudo, but it might be to change some permissions to fix this. The Python sample is even easier to use:

It’s much more fun to use the board with some I2C sensors (up to 5 are supported through the headers on the add-on board), andLeon send me a BMP180 temperature and pressure sensor, and I also connected a Grove module (accelerometer) I got from Wio Link Starter Kit. i2cdetect had no problems detecting both:

0x77 address is used by BMP180 sensor, while 0x4c address is for the grove module.

So now that we’ve made sure the BMP180 sensor has been detected we can try the sample:

The reported temperature matches the actual temperature in my room.

I initially planned to write a sample demo control my aircon using the button and the temperature sensor, so I also had to configure LIRC both to capture my aircon remote control codes, and send back the codes though the IR transmitter.

Again this is very well explained in the user’s guide, and I started by installed LIRC.

There’s no support in raspi-config for LIRC, so we have to edit /etc/modules and add the IR pins by adding the following lines:

We also have to changed four lines in /etc/lirc/hardware.conf:

and finally I had to edit  /boot/config.txt to add lirc support to dtoverlay:

Configure is done, and we can restart the Raspberry Pi board to make sure the changes are applied:

Now I’m going to capture key code from my aircon remote. First, we need to stop the service, and list of available remote key names:

Now in theory I can assign remote codes to the actual output from my aircon remote, and the idea was to use KEY_POWER for the remote power button, and KEY_DOWN to set the temperature to 25 C with the command:

Sadly, maybe one out of 25 key presses from my remote control were detected. Maybe an issue with the protocol used or timing, but I found out that I had no such problem with my TV remote control, and could complete the setup:

I configured three keys:

That means that my test/demo project had now become rather silly, as instead of turning on my aircon when it gets hot, I’d turn on the TV 🙂 But I guess it’s good enough for a review, and as a learning experience.

Now we can backup the old lirc config, replace it with ours, and restart LIRC daemon:

I could also confirm I could turn on and off the TV with my Raspberry Pi 2 and RabbitMax Flex board using the following command:

From there, it was quite straightforward to write my “useless TV demo” based on code from the samples that turns on and off the TV whenever I release the push button, or when the temperature crosses 30 Celsius, and showing the power status and temperature on the LCD display:

It works pretty well, as you can see from the video.

You’ll also find the demo code on github.

Beside Raspbian, Leon is also working on “RabbitMax IoT GNU/Linux Distribution” built with the Yocto Project where all hardware configuration is done, running an MQTT server, as well as an GHTML8 web interface designed with jQuery Mobile and Node.js API.

You can get the source code for that, as well as the documentation, C & Python sample projects, tools, and later on KiCAD files on RabbitMax github’s account, as well as some extra info on RabbitMax.com website. You can purchase the board now for $49.90 on Tindie.com without the LCD nor sensors, but it might be a good idea to wait for the crowdfunding campaign that should start shortly, with the board offered for half the Tindie price, and probably some kits with LCD, and sensors.

How to Write ESP8266 Firmware from Scratch (using ESP Bare Metal SDK and C Language)

October 7th, 2016 8 comments

CNXSoft: This is a guest post by Alexander Alashkin, software engineer in Cesanta, working on Mongoose Embedded Web Server.

Espressif’s ESP8266 had quite an evolution. Some may even call it controversial. It all started with ESP8266 being a WiFi module with a basic UART interface. But later it became clear that it’s powerful enough for embedded system. It’s essentially a module that can be used for running full-fledged applications.

esp8266-bare-metal-sdk

Espressif realized this as well and released an SDK. As first versions go, it was full of bugs but since has become significantly better. Another SDK was released which offered FreeRTOS ported to ESP. Here, I want to talk about the non-OS version. Of course, there are third-party firmwares which offer support for script language to simplify development (just Google for these), but ESP8266 is still a microchip (emphasis on MICRO) and using script language might be overkill. So what we are going to come back to is the ESP SDK and bare C. You’ll be surprised, it’s easier than it looks!

First steps

To develop firmware you’ll need:

  1. ESP8266 connected to your computer via USB.
    There are a lot of articles how to connect an ESP to a computer. You will need several Dupont cables and a UART-to-USB adapter. If you have a Arduino board you can use it as UART-to-USB. Google “connect esp8266 to computer” – there are a lot of articles about this.
  1. SDK.  I suggest using this one: https://github.com/pfalcon/esp-open-sdk

Download it and follow its readme to build. There is nothing extraordinary in this process, all you need is to install prerequisites and invoke “make”.

In general, this SDK is intended for *nix systems, but there is a port for Windows as well.

In short, to start development you should have an ESP device available as /dev/ttyUSB0 (/dev/ttyACM0 if you use Arduino or COMn in Windows) and the SDK installed in a certain path.

main()

In C int main() is an entry point to a program. But, in the case of ESP the entry point is void user_init(). This function must be used only for initialization, not for long-running logic.

Here an example:

Note, that all we do in user_init is calling the system_init_done_cb API function. This function accepts one parameter, which is a pointer to function which will be called once all system modules will be properly initialized. You can put your initialization code in user_init too, but you can face problems with some system function (like WiFi), just because appropriate modules aren’t initialized yet. Thus, it is better to use system_init_done_cb and perform initialization in the callback function.

Beware of the dog

ESP8266 has a watchdog functionality. And there is NO documented API to control it (there is some undocumented stuff, but out of scope for this tutorial). Its timeout is 1 second.

What does that mean? It means, that you have to return the control flow to system every second, otherwise the device will be rebooted. This code leads to the device reboot:

In general, watchdog is not evil, it helps if the program hangs. And, 1 second is not so small as it sounds. Just keep this fact in mind.

Doing something

Taking what we learned about the watchdog into account, we face an obvious question: where can I run my tasks?

The simplest answer is in timers. The timer API is very simple in ESP.

If the last parameter of the os_time_arm function is 0, the timer callback will be invoked only once. If it’s 1, it will be called repeatedly until the os_timer_disarm is called.

And, finally, we have a place to put our code: The start_timer_cb function.

Our task here is to make an LED blink. Some ESP boards have an LED attached to GPIO16, if your board doesn’t have it, you can attach an LED to any free GPIO.

As you remember start_timer_cb is a timer callback function, and it is called every 5 seconds. On first call on variable is 0 and we set GPIO16 to high – as result LED will be turned on. On second call we set GPIO16 to low – and LED is turned off. And so on and so on.

Building the project

Now it is time to build our project. Let’s say, we have only one source file – main.c. I cannot recommend using makefiles which are used for building examples. They are too complicated and a bit weird. So, I’d suggest to write your own (simple!) makefile.

Here are steps:

  1. Compile main.c to main.o.
    Use xtensa-lx106-elf-gcc compiler which is a part of esp-open-sdk.
  2. Link project.
    Linker to use – the same xtensa-lx106-elf-gcc. Libraries to link with are: c gcc hal m pp phy net80211 wpa main

Also, you need to supply the linker script (.ld file). Choose one from esp-open-sdk that matches the flash size of your device. After this step you’ll have .elf file.

  1. Convert .elf file to .bin

For this, use esptool.py script from esp-open-sdk. Run it like this:

If everything is ok, you should have 3 files in <output dir> with names like 0x00000.bin, 0x11000.bin 0x66000.bin.

Flashing

The final step is to put our firmware onto the device. For this we will use the esptool again, but now we should use write_flash option. Like this:

You should use real filenames from the previous step. And, if everything is still OK, the LED attached to the device will start to blink every 5 seconds.

Next steps

Writing the firmware for any device is a huge topic. Working with ESP8266 is no exception. So, the purpose of this article is only to highlight the direction. There are a lot of different APIs in the ESP8266 SDK: WiFi, GPIO, TCP/UDP and more. Make sure to check out the documentation fully here. It’s also good to check out the examples by firmware providers and esp-open-sdk. If you want to start with an example, check out this one which goes through running Mongoose Embedded Web Server on ESP8266.

Xtream Codes IPTV Panel Review – Part 2: Movie Data Editing, Security, Resellers, Users and Pricing Management

October 4th, 2016 5 comments

Dear readers, after part 1 of Xtream Codes Panel v.2.2.0 EVO review, here is part 2. I tried hard to get all in two parts, but “to be exhaustive” there will also have to be a part 3…

Movie Editing Section

xtream-codes-movie-editing

As we can see, it’s only possible to assign a movie into a single category, a SELECT BOUQUET Option under the Category would be more then useful. So while Movie Editor Section is useful right now, it still needs improvement.

Also taking each movie by hand for editing, after, for example, a Main server crash or changing the Main, is really something, a waste of time. Some of the issues / possible improvements include:

  • No mass edit of movies to bouquets
  • If changing the category of a movie, I found no working method to change in a bulk.. Even if I tried to delete the “content” of a bouquet, it was denied.
  • Left Hand Section, Mass Edit Streams allows only live streams to be edited.
  • Stream Tools! To be able to delete the falsely assigned movies. With the result of the complete addition process later on, and the transcoding process has to be repeated all over again…
  • I strongly advice customers/user’s of the Xtream Codes Panel to name their movies in a kind of unique, retrievable way later on. If not, and if you try to sort them later by assigning them in to a bouquets, you stand in front of all the movies, the way you loaded them up. No kind of Movie editing (re-naming) possible, before you load them to transcode. The mass edit movie section is simply missing.
  • This great piece of Software NEEDS A MOVIE EDITING SECTOR

The options available right now:xtream-codes-movie-editing-options

xtream-codes-movie-editing-subtitles-transcoding

  • Manage Categories has to be more specific to meet the needs of the customer.
  • Right now I was not able to really sort the categories to my preferred sequences. Did I miss here options? I couldn’t find it.
  • Edit Category function is limited to change the name only.

xtream-codes-movie-editing-category

  • Again, if you have already tons of movies assigned to a category there’s no other way to assign them in a bulk right now. A time waster.

General Warnings

To be on the safe side, do not fill your HDD up to the limit. Max 70% should give more stability.

Attention: If you rent a dedicated server, seeing the offer for example, let’s say 2x 2TB, you should know that after a normal install, you’ll only get 2TB. That is because one is mirrored, in case you loose the first partition, it would be fatal, right?

So, better let it this way, do not use the whole size. Agreed?

Having issues with reconnecting interrupted streams, I filled Xtream Codes panel with over 200 streams, and around 500 movies. I checked the stopped streams, and most connected again after manually pressing the “start/restart stream” button. But these are rare cases only, overall I must say the platform is absolutely stable.

I found a few times the ‘Guilty-ONE” for this: It was not Xtream Codes Panel, it was the miserable Streams I got from a friend for testing purposes.

Security settings (Fight the Intruders by blocking them)

Handle this setting carefully! Especially if you give a trial List to a customer, he will zapp through the channels quickly, or VLC crash often, how we all know..

Wrong or too hard settings will block him immediately.

A good setting should be as follows:

  1. General Settings->Security Settings->Flood Limit to 6 or 8
  2. Request Frequency in Seconds to 1.

Security Settings Section

Back to the Security “plugins”

xtream-codes-security-pluginsBLOCK IP/CIDR

xtream-codes-security-block-ip-cidr

We see the IP’s or better, the intruders (hint: After you set-up the Flood Limits in General-Security/Settings)

Here we go, nice to look at… but why not show directly the “customer by name with IP”?. In the end, we have to make the decision whether to unlock the IP or keep him locked. It’s only working if a server with a fixed IP started the attack, dynamic IP’s appear soon then if the IP changed.

And in case no internal customer, let’s say a simple “from outside” or no customer would be a great help. Going through the log files  to find out if it us a customer or instead a competitor trying to slow down your server (we are all nosy, aren’t we?) makes no sense and no fun. Reason: time extensive if dozens or hundreds of logs!

Live Streams-Video on Demand Section

Manage Stream/Add New Stream

xtream-codes-live-streams-vodThere is also one relatively “unique” function; you should handle with care, or best ask the seller of the LEGITIMATE streams, if he allows that you use the “Use Direct Source & don’t restream it”. This means all coming traffics are not on your side, the connected server takes it all. You should be able to see the other side. But I guess you know already.

You can also redirect the Stream to the Original Source, so that your customers connect through your own DNS. Doing so, setting to yes, your URL will show up in the Bouquet List later. But again, be aware, choosing NO, your user will not connect to your server in any case. I tested it by letting the user connect over my DNS. I thought, I would see any client connecting through my DNS, but you will not, in both cases. No matter Client Connection Logs or in Manage Lines, he is simply invisible to you. I guess, after all not the smartest idea.

xtream-codes-dns

Security Section

General settings->Security Settings

xtream-codes-security-settings

Useful if you find the “right” setting. The Flood Limit shows the number of requests specified in “Request Frequency”. I started with 5 requests within 3 seconds, result: I got blocked myself and no idea why. I just watched one channel, no fast channel switching, no nothing. This seems to getting interesting, I thought.

Switched to 8 requests in 3 seconds range. Activated my test users… Result: 2 Android boxes getting blocked.

Mhhhh, I go deeper in this then after, and because of English is not my native language, haha, we “none-english” have sometimes a problem to understand the language. And…I found it! My mistake! Definitely to 90% only, haha! And 10% I give to Xtream Codes back, because they should really be more specific about this classy feature, which is highly useful!

Advice: Give 3 options, or 4, the ordinary customer can use! Like: LOW-Standard-HIGH-HYPER Settings, pre-configured! (Also the option for manual settings).

I know I know…but remember this: Many customers are not common with these terms; they simply do mistakes resulting in blocking customers who do nothing wrong at all, which could lead to losing customers.

Standard should be: 10:1, or 20:1 ((flood limit:request frequency), but hard to say for sure in my opinion. And even this could maybe cause some troubles, if a nervous user zapping the channel’s in a lightspeed and the playlist is huge…

For example: Each zapping under 1 second is 1 evil request, collected and memorized by the panel! 10:1  means, 10 times under 1 second is OK, but then… the 11th try would blocked them! This is a hard decision to make for each panel user, if he has a huge playlist with many customers. Some customers could complain, if they get blocked, and I don’t want to talk about the non-stop checking the Security Settings, and look for blocked IPs.

Recommendation: Invest in a DDOS Protection with your server, every hosting provider is offering this. Also let this setting to OFF (0) in the beginning, and get used to everything first. You can then implement the settings you are comfortable with lateron.

Resellers – Registered Users Section

xtream-codes-users-resellers

Reseller Section

Registered Users and Reseller

When I started to work with the panel and getting used to it, this was the first category I started with. I expected something like: REGISTER RESELLER, or similar. Registered means already already registered, right?

Let’s start with “Register New User”. This feature is mainly used, to create a new RESELLER of yours, a now you want to give him access to a reseller panel. Let’s create one!

xtream-codes-register-new-user

As we can see, you have to provide the follwing details for the new user (reseller in this case):

  • Username (Whatever you want to call the reseller)
  • Password (Please 6 digits minimum, out of security reasons, phrases are also useful)
  • E-Mail (Email address of your reseller, without one you can not create a new reseller)
  • Default Language (right now only English is possible)
  • Group Members (see the next Section, Group Members)
  • Reseller DNS (Reseller Only) – Here you can specify the reseller’s own DNSs
  • Credits (reseller Only) – The amount of money the Reseller payed you, usual is 1 credit = 1 UK pound or other currencies)

Section Manage Group Members

xtream-codes-create-new-group
First of all, a little explanation here on this important part. As we’ve already seen above, we can “register” a new user, reseller in this case with “REGISTER NEW USER” option (I would rather see a “Create new Reseller” button). But here is the point: If you register a new future reseller, you have to assign the payment formalities later (ex: 300 Euro = 300Credits), but also how much for 1 month, 3 months or any other kind of subscription (time frame), and the reseller can create lines for his own customers later, and see his own costs each time.

Assigning them (the resellers) all in one category, named reseller, let you later no choice to give different kind of subscription prices. (ex: Reseller A 1 month =6 Euro, Reseller B 1 month 8 Euro), so that’s why Xtream Codes has a great feature in creating groups.

You can simply create a new “group”, let’s say Group elcap in my case, and assign reseller A to it. Now your Reseller A is a Group Member of “elcap” group. Next step, you want to give him prices and time frames of each line he creates.

xtream-codes-price-options

As you can see, I created the trials , 1 month, 3 months, 6 months and 12 months, and also a VOD package for the reseller part of group elcap.

You start with Resellers -> New Package (Adds a new Package)

xtream-codes-resellersWe see the following options after clicking on New Package:xtream-codes-resellers-new-package-1

xtream-codes-resellers-new-package-2

You first need to name the package, mark it as Trial or Official Package (1-12 month or else later, you are free in your decision), you can even pair the line with the ISP provider of your client by first connection. (Attn.: need additional plugin to buy).

Useful only if you create a Package for a RESTREAMER (Restreamers are able to put your bouquets/packages on their server(s) and use them as there own streams), as they have a fixed IP in any case.

In this example we give 20 connections to a Restreamer

xtream-codes-resellers-restreamer

Meaning: You can give your whole LEGITIMATE stream list, or assorted ones, like a bouquet you filled with specific streams before. In any case, your Restreamer you are selling to, is only able to open 20 connections on his server. The amount of connections you assign to him is what he pays for.

Settings Prices with Manage Packages

xtream-codes-resellers-manage-package-price

Edit a packagextream-codes-resellers-edit-package

xtream-codes-resellers-edit-package-2

A complete option to edit your before assigned package (Manage Package->Edit Package).

Here’s the output of my own created packages Create them for your own needs, everything is possible!

xtream-codes-subcription-examples

Manage ROOT (Registered Users)

xtream-codes-registered-users

By clicking on Manage Root you get the following options:

xtream-codes-users-manage-root

Here you can change your ROOT username/password given by Xtream Codes for your panel.

You can change both passwords here later, if you do not wish to use the old ones once given from Xtream Codes after you ordered and paid for your panel. Handle this carefully, write it down and copy it to safe locations, just in case of a crash of the main server (see upcoming part 3, with simulated crash of the Main server)

I initially planned to finish this review into two parts, but there are so many options, I have to do at least one more part.

I also have to repeat Options and Features for better understanding later on. See you guys on Part 3 soon, which should include:

  1. Changing the Main Server (through my extended kind of tests, I worked a couple of days on this, and the results should benefit all of us who are using this software and working with it.)
  2. Finishing the features for good, I promise
  3. The “Have’s” and the “Have Not’s
  4. Enumerations of the Top 5 settings to use
  5. Results after a new installation (Simulated Crash of the Main Server)
  6. The Pro’s & the contra’s
  7. The Competitors of Xtreme-Codes Panel
  8. Be aware of some kind of hosting services (Strange experiences with 2 “Big Ones”! Troubleshootings)
  9. The Conclusion
  10. Preparing a little Manual (PDF format, to help you guys)

[Update: During finishing my work on the huge Part 3 of the Review Xtream Codes Panel Version.2.2.0 EVO, there was yesterday, 08th October, an update to Version 2.3.0 EV0, with a lot of new Features and Improvements!

So, I will need a little longer to finish the Part 3, please stay tuned.]

How to check HTTP Header and Connection Stats from the Command Line

October 3rd, 2016 1 comment

A few days ago, I discussed with somebody whether a file was cached by Cloudflare or not, and this involved getting the HTTP header, and checking for CF-RAY field to see if data is going through one of Cloudflare data centers. This can be done with curl:

In the command above, -s stands for silent so that curl does not show the progress meter, -v stands for verbose to show the header, and -o /dev/null is used to discard the packet load.

You can also use -I option (fetch the HTTP-header only) with curl, which – if all you need is the HTTP header – provides a cleaner output:

I also came across httpstat Python script recently via n0where, doing much of the same thing, except it also adds transfer statistics.
It can be installed by downloading httpstat.py, or better using pip:

Let’s try it with this very blog:

The header is the same as with Curl -I, and for good reason, since httpstat relies on curl, and all curl options (except for -w, -D, -o, -s, -S) can be used with the command line. This script can be useful if you are trying to benchmark and lower TTFB (Time To First Byte), or decrease overall download times from your server.

httpstat

How to Download YouTube 4K Videos with Youtube-dl Script

September 20th, 2016 7 comments

Many Android devices can now support 4K VP9 and/or H.264 video decoding, but for some reasons, Google only serves 4K videos to devices running Android TV OS, as opposed to just Android, with the latter limited to 1080p videos. If for some reasons you want to watch a YouTube 4K video on a capable device, you could download it with youtube-dl script written in Python, and available for Linux, Windows, and Mac OS.

youtube-4k-video-downloadI’ve been using the script for a while to download various video or audio files from YouTube, and other website, but for the purpose of this post, I made sure to update it to the latest version:

If you have already installed the script, it can also be updated with:

I’ll use one of the most popular 4K videos on YouTube as example, namely: COSTA RICA IN 4K 60fps (ULTRA HD) w/ Freefly Movi.

The first task is to list the video’s supported formats:

So we can see there are lots of different formats for a given video with different resolution, container formats, and bitrates. You have to find the lines you want and select the corresponding “format code” in the first column to download the video and audio. I’ll select 313 for video and let the script select the best audio format:

The “+” sign is used to merge video and audio. If you don’t add an option for audio, only the video without audio will be downloaded. You can also run “youtube-dl URL” without options, and it should download the video with both “best audio and video”, but you don’t have full control over the exact format of the video.

Youtube-hd script has many others options, include support for YouTube playlist downloads so it’s a very useful script.

Categories: Linux, Testing Tags: 4k, how-to, tutorial, youtube