A Fine Slice Of SQL Server
Suppose we had a large data set hosted on a
Sparkcluster that we wished to work with usingdplyrandsparklyr(for this article we will simulate such using data loaded intoSparkfrom thenycflights13package).We will work a trivial example: taking a quick peek at your data. The analyst should always be able to and willing to look at the data.
It is easy to look at the top of the data, or any specific set of rows of the data.
Read on for more details.
Comments closedJiang Mouren has a two-parter on WebHCat. First, how it works:
SSH shell/Oozie hive action directly interact with YARN for HIVE execution where as Program using HdInsight Jobs SDK/ADF (Azure Data Factory) uses WebHCat REST interface to submit the jobs.
WebHCat is a REST interface for remote jobs (Hive, Pig, Scoop, MapReduce) execution. WebHCat translates the job submission requests into YARN applications and reports the status based on the YARN application status. WebHCat results are coming from YARN and troubleshooting some of them needs to go to YARN.
2.1.2. WebHCat times out
HDInsight Gateway times out responses which take longer than 2Minutes resulting in “502 BadGateway”. WebHCat queries YARN services for job status and if they take longer than the request might timeout.
When this happens collect the following logs for further investigation:
/var/log/webchat. Typical contents of directory will be like
- webhcat.log is the log4j log to which server writes logs
- webhcat-console.log is stdout of server is started.
- webhcat-console-error.log is stderr of server process
NOTE: webhcat.log will roll-over daily hence files like webhcat.log.YYYY-MM-DD will also present. For logs to a specific time range make sure that appropriate file is selected.
Because HDInsight doesn’t support WebHDFS, WebHCat is the primary method for cluster access, so it’s good to know.
Comments closedRanjan Banerjee explains how the Java garbage collector works, using Hadoop as an example:
The reason why we all love Java is due to the fact that we can be careless with memory creations and the work of cleaning the mess is performed by the JVM. On a high level, Java heap memory is classified into two phases:
1) Young (eden) space
2)Old space
The eden space is where newly created objects goto. There are various algorithms for garbage collection, but all of them try to first free memory from the young space and for those long lasting memory objects, they are transferred to the old space.
One common issue that can be noticed in running Map Reduce Applications are GC overhead limit exceeded.
Read on for more, including where you can find GC logs.
Comments closedVinod Kumar Vavilapalli describes the modern data lake:
During the past few years though, end-to-end business use-cases have evolved to another level.
- The end-to-end business problems are now mostly solved by multiple applications working together.
- As the platform matured, users have increasingly started wanting to solely focus on the business application layers, and getting impatient to get on with developing their main business-logic.
- However, YARN, and for that matter any other related platform, hasn’t catered to this evolving need, leaving the users to unwillingly get involved in the painstaking details of wiring applications together, keeping them up, manually scaling them as need arises etc.
Manual plumbing of all these different colored services in tiresome! Further, there is a clear need for seamless aggregate deployment, lifecycle management and application wireup. This is the gap that needs to be bridged between what these end-to-end business use-cases need from the platform and what the platform offers today. If these features are provided, then the business use cases authors can singularly focus on the business logic.
This is a higher-level “where are we at?” kind of post which could be helpful if you’re new to the data lake concept.
Comments closedTim Spann explains how to perform backup/recovery operations and disaster recovery using Hadoop:
You can mirror datasets with Falcon. Mirroring is a very useful option for enterprises and is well-documented. This is something that you may want to get validated by a third party. See the following resources:
- Hive DR with Falcon.
- Data movement and integration (this overview from Hortonworks is very useful for practical data movement between and within the cluster).
- Falcon details (in-depth presentation).
Tim shows several recovery options, making it useful reading if you use Hadoop as a source system for anything (or if you can’t afford it to be down for a 2-3 day period as you recover data).
Comments closedJitendra Bafna explains how HBase compaction works:
Compaction is a process by which HBase cleans itself. It comes in two flavors: minor compaction and major compaction.
Minor compaction is the process of combining the configurable number of smaller HFiles into one Large HFile. Minor compaction is very important because without it, reading particular rows requires many disk reads and can reduce overall performance.
Major compaction is a process of combining the StoreFiles of regions into a single StoreFile. It also deletes remove and expired versions. By default, major compaction runs every 24 hours and merges all StoreFiles into single StoreFile. After compaction, if the new larger StoreFile is greater than a certain size (defined by property), the region will split into new regions.
Read on for more information about compaction and data locality, which is a totally different topic.
Comments closedAmit Kulkarni shows how to install Azure Data Lake Store support on your “older” Hadoop clusters:
How old is really old?
The Azure Data Lake Store binaries have been broadly certified for Hadoop distributions after 3.0 and above. We are really in uncharted territory for lower versions. So the farther away you go from 3.0 the higher the likelihood of them not working. My personal recommendation is to go no lower than 2.6. After that your mileage may really vary.
This is a good article, and do check it out. A very small mini-rant follows: Hadoop version 2.6 is not old. Nor is 2.7. 2.7 is the most recent production-worthy branch and 3.0 isn’t expected to go GA until August.
Comments closedAndrew Wang and Ray Chaing note that a new Hadoop 3 alpha is now available:
YARN introduces the notion of opportunistic containers in addition to the current guaranteed containers. An opportunistic container is queued at the NodeManager waiting for resources to become available, and run opportunistically so long as resources are available. They are preempted, if and when needed, to make room for guaranteed containers. Running opportunistic containers between the completion of a guaranteed container and the allocation of a new one should improve cluster utilization.
There are a couple other new features, including support for Azure Data Lake Store.
Comments closedAmit Kulkarni shows how to make Azure Data Lake Store the default file system for a Hadoop cluster:
So to give a concrete example, if the default file system was
hdfs://123.23.12.4344:9000then the/user/filename.txtwould resolve tohdfs://123.23.12.4344:9000/user/filename.txt.Why does the default file system matter? The first answer to this is purely convenience. It is a heck lot easier to simply say
/events/sensor1/thanadl://amitadls.azuredatalakestore.net/in code and configurations. Secondly, many components in Hadoop use relative paths by default. For instance there are a fixed set of places, specified by relative paths, where various applications generate their log files. Finally, many ISV applications running on Hadoop specify important locations by relative paths.
Read on to see how.
Comments closedPeter Coates talks about pros and cons to Hadoop in the cloud:
Hadoop was developed for deployment over Linux running on bare metal. Cloud deployment implies virtual machines, and for Hadoop it’s a huge difference.
As detailed in other articles (for instance, Your Cluster Is an Appliance or Understanding Hadoop Hardware Requirements), bare-metal deployments have an inherent advantage over virtual machine deployments. The biggest of these is that they can use direct attached storage, i.e., local disks.
Not every Hadoop workload is storage I/O bound, but most are, and even when Hadoop seems to be CPU bound, much of the CPU activity is often either directly in service of I/O, i.e., marshaling, unmarshaling, compression, etc., or in service of avoiding I/O, i.e., building in-memory tables for map-side joins.
Read the whole thing.
Comments closed